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Executive Summary 


The Literacy Design Collaborative (LDC) was created to support teachers in implementing 
Common Core State Standards (CCSS) in order to teach literacy skills throughout the content 
areas. The LDC Investing in Innovation (i3) project focuses on developing teacher competencies 
through job-embedded professional development and the use of professional learning 
communities (PLCs). Teachers work collaboratively with coaches to further develop their 
expertise and design standards-driven, literacy-rich writing assignments within their existing 
curricula across all content areas. 


Engaged in the evaluation of LDC tools since June 2011, UCLA’s National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST) is the independent evaluator 
for LDC’s federally funded Investing in Innovation (i3) validation grant. The 2016-2017 school 
year was the first year of implementation, following a pilot year during which the 
implementation plan, instruments, data collection processes, and analytical methodologies 
were refined. 


This annual report presents an initial look at LDC implementation in the first cohort of 29 
schools in the New York City Department of Education (NYCDOE) during their first year of 
implementation. The early results suggest the following: 


e Participants across all groups reported positive attitudes toward LDC. All measures of 
satisfaction or improvement were rated positively by more than half of respondents. 
Two thirds of teachers expressed interest in learning more about how to lead LDC 
implementation at their schools, and over 80% of project liaisons and administrators 
anticipated that their teachers would continue with LDC the following year. 


e Participants perceive a positive impact on student outcomes. Over two thirds of 
teachers and over 90% of administrators agreed that LDC helped improve students’ 
literacy performance. 


e Individuals leading and supporting the LDC implementation at all levels received highly 
positive ratings. LDC coaches were rated by 95% of teachers as providing appropriate 


and timely feedback. Project liaisons were overwhelmingly reported to be highly 
approachable, effective, and knowledgeable. Almost all teachers reported that their 
administrators encouraged LDC participation in schools. A large majority of project 
liaisons and administrators had positive interactions with LDC staff and were able to 
receive appropriate resources and support when needed. 


e At this point, there is insufficient quantitative evidence to suggest a positive LDC 
impact on student test scores either at the elementary or middle school level. This 
finding should not be surprising given the early stage of intervention, with teachers 
having only completed one year of the two-year implementation process. 


e The LDC intervention appears to have differential results for teachers in different 
content areas. It seems to be a better fit for English language arts and history/social 
studies teachers than for science and math teachers. Teacher feedback and level of 
engagement with CoreTools indicated that science and math teachers were less 
engaged with the material. 


e This district’s implementation did, on average, appear to have met LDC’s participation 
expectations for high implementation. The ideal is that PLC members meet weekly for 
at least 60 minutes. Almost two thirds of teachers reported meeting at least once a 
week or more. Over half reported that meetings lasted 45 minutes to an hour, and 
30% reported they lasted longer than an hour. Almost three quarters of teachers 
agreed that their PLC was given sufficient time to meet, although many teachers who 
provided open-ended responses asked for more protected, paid time. 


e Analyses of module artifacts suggest that both elementary and secondary teachers 
were moderately successful at creating an effective writing task. Secondary teachers 
were moderately successful at aligning to standards. Modules did not reflect success 
in fidelity to LDC, quality instructional strategies, coherence and clarity, and overall 
impression. 


As an ongoing multiyear intervention, the LDC implementation will continue to evolve 
year to year as participants provide feedback and LDC program managers make refinements. 
Thus, we anticipate that further significant changes to the course material and the delivery 
system that are already in progress for Year 2 will likely result in continued and possibly 
increased positive feedback. Related, we posit that further support for science and math 
teachers would likely result in higher levels of success and satisfaction for those teachers. 
Finally, as teachers return for a second year and achieve greater experience with the LDC 
model, it is likely that their ability to apply their learning in increasingly productive ways will 
become more evident in their self-reports, module quality, and engagement with the LDC 
platform. 
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1.0 Introduction 


The Literacy Design Collaborative (LDC) was created to support teachers in implementing 
Common Core State Standards (CCSS) in order to teach literacy skills throughout the content 
areas. The LDC Investing in Innovation (i3) project focuses on developing teacher competencies 
through job-embedded professional development and the use of professional learning 
communities (PLCs). Teachers work collaboratively with coaches to further develop their 
expertise and design standards-driven, literacy-rich writing assignments within their existing 
curriculum across all content areas. LDC is a national community of educators providing a 
teacher-designed and research-based framework, online tools, and resources for creating both 
literacy-rich assignments and courses across content areas. Used by individual teachers, 
schools, and districts in 40 states for the past four years, LDC also is a statewide adopted 
strategy for Common Core implementation in Kentucky, Colorado, Louisiana, and Georgia. 


UCLA’s National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), in collaboration with its partner Research for Action (RFA), engaged in the evaluation 
of the implementation and impact of LDC tools on student learning and teacher effectiveness 
starting in June 2011, via two parallel research studies funded by the Bill and Melinda Gates 
Foundation. Those studies included an examination of LDC implementation in eighth grade 
social studies and science classrooms in Kentucky and Pennsylvania and a districtwide 
implementation in sixth grade advanced reading classes in a large district in Florida. Results for 
the studies are available in two technical reports (Herman et al., 2015a; Herman et al., 2015b), 
as well as a journal article published by AERA Open (Herman, Epstein, & Leon, 2016). 


Currently, CRESST serves as the independent evaluator for LDC’s federally funded 
Investing in Innovation (i3) validation grant. The LDC i3 study is examining the implementation 
and impact of LDC in two large school districts: New York City Department of Education and a 
large school district on the West Coast. The evaluation study is a comprehensive mixed- 
methods evaluation to understand the impact of LDC on student learning using a quasi- 
experimental design, as well as to document impact on teacher skills and practices. Specifically, 


the evaluation study addresses a rich range of questions about program characteristics, 
conditions, and program impacts in the context of two large urban school districts. The study 
will draw on data from two cohorts of schools, with each school housing a professional learning 
community (PLC) of teachers engaging in professional learning about LDC and implementing 
LDC mini-tasks and modules in their classrooms. We will measure teacher implementation and 
skill improvement via teacher surveys, analysis of analytic data from LDC’s online CoreTools 
module building platform, and artifact analysis. While we will document the core strategies of 
the LDC model as implemented and provide support for LDC improvement, the central focus of 
our comprehensive mixed-methods evaluation is examining the impact of LDC on teacher 
practices and student learning using a quasi-experimental design. 


The first i3 evaluation cohort of schools began implementing LDC during the 2016-2017 
school year. This annual progress report examines LDC implementation during the 2016-2017 
school year in the New York City Department of Education (NYCDOE). A parallel progress report 
focuses on implementation in a large district on the West Coast. The current annual progress 
report presents results from (a) analyses describing how LDC participants interacted with the 
CoreTools module building platform; (b) scoring by CRESST of instructional modules created by 
LDC participants; (c) surveys of classroom teachers, LDC project liaisons, and school 
administrators; and (d) student outcome analyses using the quasi-experimental design. 


These results provide a window into how LDC was implemented in 2016-2017, the 
perceived utility and effectiveness of various program components, and the perceived impact 
of LDC on both teacher and student skills and knowledge. A preliminary test of the 
effectiveness of LDC in increasing student learning is also included in the report. 


1.1 Logic Model 


The logic model includes four key intervention components that were predicted to be the 
drivers of change in teacher practice and student learning (see Figure 1.1). These are a coach- 
supported Professional Learning Community (PLC) formed to implement the LDC intervention 
at the school site and provide a space for teacher collaboration; asynchronous support from 
coaches in the form of feedback in CoreTools through comments and peer review; 
implementation activities completed by participating teachers including module development 
and classroom implementation; and leadership support at different levels. Note that the model 
also indicates LDC’s implementation expectations in each area. 


LITERACY DESIGN COLLABORATIVE i3 LOGIC MODEL 


Participation Short term | | Medium term 


Common Planni Student Increased Students 
time for LDC ng - PLC members participate in at least 60 engagement CCRS skills acquire 


minutes of common planning time during 


Professional knowledge and 


Learning each consecutive two weeks. Improved skills to be 
Community (PLC) Digital coach joins PLC every other week test scores college and 
with synchronous Sequencing incorporates blocks from LDC career ready 
coach support online courses. Higher rates 

Increased teacher of course Educational 

expertise and skill completion attainment 
Asynchronous development increases 
support from LDC Coach provides feedback on modules in ae 
coaches CoreTools via comments and peer review igher 

rubric ratings. ection 


Increased labor 
market 


Teacher - Teachers develop at least one module per outcomes 
implementation year in Years 1-2 
activities - Teachers teach at least two modules per year 

in Years 1-3. 


More effective 


CCRS-aligned 
instruction 
District instructional specialists visit schools and use of 
three times per year to sit inon PLC, meet with formative 
Leadership principal and teacher leader, and observe assessment 
support at instruction. 
different levels - Principals participate in LDC by protecting PLC 


time, setting goals for LDC work, facilitating & 
participating in schoo! visits from district 
instructional specialists, and observing LDC 


instruction 


Teacher leaders facilitate PLC on the ground, 
collaborate with administrator, and 
communicate regularly with coach. 


Contextual Factors: District support, school and student demographics, other policies and initiatives 


Figure 1.1. LDC i3 logic model. 


The logic model predicts that the four key components will lead to increased teacher 
expertise and skill development and more effective Common Core aligned instruction which 
incorporates ongoing formative assessment. In turn, increased teacher capacity and more 
effective instruction will lead to increased student engagement in the short term; increased 
student skill acquisition, higher test scores, and higher rates of course completion in the 
medium term; and improved college and career readiness, education attainment, graduation 
rates, and labor market outcomes in the long term. 


Note that the logic model has been revised based on refinements to the program in 
response to learning from the pilot year (2015-2016) and the first year for implementation 
cohort 1 (2016-2017). The logic model presented here is current as of winter 2018. 


Note also that Figure 1.1 refers to teacher leaders, but this report will refer to project 
liaisons. That distinction reflects an update to the model; starting in 2017-2018, teacher 
leaders are being identified in the first year that a school implements LDC, and those teacher 
leaders will receive a stipend in their first year. This change was not yet in effect during the 


2016-2017 school year, and we therefore refer to teachers playing a leadership role in LDC as 
project liaisons in this report. 


1.2 Evaluation Questions 


Our evaluation questions focus on addressing three main areas: program characteristics 
and implementation, contextual factors and implementation, and program impacts. This 
progress report provides findings on many, but not all, of the evaluation questions. In 
particular, given that the evaluation is still in its early stages, there is limited information 
available regarding program impacts. This report provides a first exploratory look at how the 
refined LDC model is impacting student learning. The first confirmatory analysis will be 
conducted at the end of two years of participation for the teachers. 


1. Program Characteristics and Implementation 


a. 


Who are the participating teachers and schools? Are they representative of the 
teacher/school populations of the respective district on years of teaching, 
education level, prior student performance, etc.? 


How is the LDC program implemented in each district? What are the core 
components (e.g., training, tools, on-site or other direct support) and who are the 
key participants? In what ways did the LDC implementation align with the 
intended model? 


In what ways do teachers implement the LDC tools in their classrooms? To what 
extent do teacher practices align with intended LDC practices? 


How are teachers utilizing the online LDC system (including online tools, 
exemplars, collaborative work spaces, and technical assistance) in terms of 
frequency and use of key features? Does this vary by teacher characteristics? What 
are teachers’ perceptions of the value and quality of the online LDC system? 


What types of LDC professional development opportunities are offered to and 
utilized by teachers at each school/district? Are teachers and schools satisfied with 
the LDC professional development opportunities they received? 


2. Contextual Factors and Implementation 


a. 


What factors facilitate or hinder successful implementation of the LDC model at 
the teacher, school, and district levels? 


How can implementation of the model be improved at the teacher, school, and 
district levels? 


What other educational reforms are being implemented in the participating 
schools and districts? What are their influences on the LDC adoption in the schools 
and districts? Are schools able to align reform efforts? 


What are the roles of school and district leadership in shaping the LDC 
implementation? 


3. Program Impacts 


a. 


What is the impact of LDC on the academic performance of participating students 
as measured by the state assessments? 


Do the academic impacts vary by student subgroup including prior achievement, 
race, ethnicity, socioeconomic status, gender, language proficiency, and/or 
disability? Does LDC help close the achievement gap between student subgroups? 


Do the academic impacts vary by student grade level or subject? 


What is the impact of LDC on teacher skill improvement and learning as measured 
by CoreTools and by the quality of LDC modules they produce? What is the self- 
reported impact of LDC on teacher learning? 


To what extent do teachers report changes in their practice (e.g., teaching 
strategy, collaboration with others) and changes in their comfort in implementing 
CCSS during and after the LDC intervention? 


What is the relationship between the fidelity of implementation, fidelity of 
intervention, and student learning? What are the conditions and contexts under 
which LDC tool use is most effective? 


To what extent do Cohort 1 participating schools and teachers continue their LDC- 
influenced practices in the 2019-2020 school year after the LDC support ends? 
What contributed to their decision to continue or stop? What factors contributed 
to their levels of continued implementation? How does Cohort 1’s actions align 
with their previously stated intentions for continuation of LDC-influenced practices 
as reported in spring 2017? To what extent do Cohort 2 participating schools and 
teachers plan to continue their LDC-influenced practices after the LDC support 
ends? 


2.0 Study Methodology 


In this chapter we provide an overview of the methodology behind this early look at LDC 
in 2016-2017. We begin by describing the various instruments and data sources for the 
analyses, including (a) analytic data from LDC’s CoreTools platform; (b) module artifacts 
including samples of student work; (c) surveys of classroom teachers and project liaisons 
participating in PLCs and administrators overseeing the implementation; and (d) administrative 
data on students and teachers used for outcomes analyses. We then describe the sample of 
educators and schools for each of these data sources. Finally, we discuss the methodological 
approaches for the various analyses we conducted. 


2.1 Data and Instruments 


We describe below each of the data instruments and the elements they contain. Most 
variables are measured at the teacher level, which is the unit at which the LDC intervention is 
being implemented. Administrative data for the analysis of the impact of LDC on student 
learning will include school-, teacher-, and student-level variables. 


LDC CoreTools. The CRESST team received the LDC program data on i3 participants’ 
interactions with the CoreTools module building platform. The data files captured three key 
activities related to the module building platform: document page viewing, document editing, 
and document commenting. 


Specifically, the data contained date- and time-stamped records of participants’ activities 
in each of these areas, and we analyzed variation in the number of times the participants 
performed these activities across the school year. We generated descriptive statistics 
(minimum, maximum, mean, standard deviation) for the number of times participants viewed a 
document page, edited a module document, and commented on a module document. We also 
produced descriptive statistics on these behaviors for various roles (teacher, project liaison, 
administrator), school level (elementary, middle, high) and content area subgroups. Finally, we 
examined the difference in average engagement in these key activities between teachers 
whose completed modules we rated in Chapter 4 of this report and those teachers who did not 
complete a module. 


Modules. Our existing rubrics, developed for our prior LDC evaluation work (Herman et 
al., 2015a), were adapted to examine the quality and coherence of the LDC modules and to 
address the quality of both content and literacy development materials (e.g., template task, 
student work samples, and descriptions of the pacing and goals of the modules). The six 
dimensions examined for this study included the following: (1) effective writing task; (2) 
alignment to CCSS and local and state literacy and content standards; (3) fidelity to LDC module 


1See Reisman, Herman, Luskin, and Epstein (2013) for a summary of the original generalizability study conducted 
using the CRESST-developed rubrics. We excluded three dimensions that focused on issues of text quality as texts 
selected by teachers were not readily available in CoreTools for the analysis. 


instruction; (4) quality of instructional strategies; (5) coherence and clarity of module; and (6) 
overall impression. Three additional dimensions that focused on issues of text quality were 
excluded since submissions did not include copies of the materials used by the teachers. Each 
dimension was rated using a 5-point scale with anchor points for the first five dimensions 
ranging from not present or not realized to fully present or fully realized and anchor points for 
the final dimension ranging from inadequate to advanced LDC module implementation. 
Detailed definitions of each dimension and descriptions for what constitutes ratings of 1, 3, and 
5 on each dimension can be found in the rubric in Appendix A. 


Surveys (teachers, project liaisons, and administrators). In collaboration with LDC, 
CRESST made substantial revisions to pilot year (2015-2016) surveys. Revisions address lessons 
learned from administration and analysis of the pilot surveys, adjustments to the program 
model made during and subsequent to the pilot year, and a desire to yield more robust 
information on teacher skills and practices. Items were also added to help understand in which 
grades and classes teachers were implementing LDC and to help identify the modules teachers 
were spending their time and energy on. Similar to the pilot year, CRESST designed five surveys 
to capture data on the experience of LDC participants playing three different roles: teacher, 
project liaison, and administrator. Some project liaisons were also administrators or teachers. 
Thus, five versions of the surveys were administered in spring 2017: (1) teacher, (2) 
teacher/project liaison, (3) project liaison, (4) administrator, and (5) administrator/project 
liaison. 


The surveys were designed to capture multiple perspectives on key aspects of LDC’s logic 
model? (see Figure 1.1), and to provide data to answer the evaluation’s research questions 
presented earlier. Survey questions targeted at the three roles fall under the domains and 
subdomains in Table 2.1. Domains were selected to align with the LDC i3 logic model and with 
the CRESST evaluation questions. Note that most domains cover multiple subdomains, 
constructs, and survey questions. Professional Learning Community/Teacher Collaboration, for 
example, captures the intensity, frequency, and collaborative environment of common planning 
time; LDC Training and Support includes quality of online courses, utility and effectiveness of 
coach support, etc.; and LDC Implementation encompasses module creation, classroom 
implementation of modules, and module peer review. 


*The survey domains were aligned to this version of the logic model for the pilot year. The logic model has since 
been revised to align with the revised LDC implementation plan. 


Table 2.1 


Survey Domains for Three Respondent Groups 


Domain Teacher Project liaison Administrator 

LDC Participation X X X 
Professional Learning Community and Teacher Collaboration X X X 
LDC Training and Support X X X 
LDC Implementation 

Module Creation X X 

Classroom Implementation X 

Module Peer Review X 

Alignment X X 
Leadership Support 

Project Liaison Support X 

School Administrator Support / Classroom Observation X X X 

Project Liaison Leadership Role X X X 

District Support X X 
Impact 

Impact on Teacher Practice and Learning X X 

Impact on Student Learning X X 
Scale-Up and Sustainability X X 
Facilitators and Barriers 
Areas of Improvement X X X 


Teachers and administrators were asked to reflect on both LDC’s /mpact on Teacher 
Practice and Learning and Impact on Student Learning. Questions within a number of domains 
further asked respondents to reflect on conditions and supports that may potentially impact 
LDC’s implementation. These domains included teachers’ perceptions of Facilitators and 
Barriers to implementation and perceptions regarding leadership roles and support for LDC at 
different levels. Project liaisons and administrators were also asked for their perceptions 
regarding if and how LDC would be sustained and expanded within the school. Finally, all 
respondents were asked open-ended questions regarding Areas of Improvement for LDC 
implementation. Teacher, project liaison, and administrator surveys can be found in 
Appendices B, C, and D. 


Administrative data used in student outcomes analysis. Student-level variables utilized in 
the outcomes analysis included race/ethnicity, gender, poverty status, special education status, 
English language proficiency, gifted status, grade, and prior achievement in math and ELA and 
outcome year achievement in ELA on state assessments. Teacher-level indicators obtained and 
utilized included years of teaching experience and teacher attendance. We also requested and 
received roster files that establish a link between teachers and students via specific courses. 


2.2 Sample 


Twenty-nine Cohort 1 schools began implementing the LDC program in the 2016-2017 
school year, with 217 classroom teachers participating and 48 administrators overseeing the 
work (see Table 2.2). The 29 schools included two elementary, four K-8, 17 middle, three 6-12, 
and three high schools. Participants taught across all grade levels from K to 12. Most secondary 
teachers taught ELA, social studies/history, or science, with a smaller number teaching math. 


As can be seen in Table 2.2, across the different measures data were available for a large 
majority of participants. Ninety percent of teachers consented to participate in the study, with 
85% of all teachers completing the survey in spring 2017. The consent rate (75%) and survey 
response rate (67%) for administrators were a little lower than the corresponding rates for 
teachers. The CoreTools dataset, which was delivered to CRESST directly by LDC and did not 
depend on teachers’ individual study consents, captured a slightly larger number of teachers 
and administrators than those that consented to the CRESST survey. 


In addition to the CoreTools analytic files, we also received module artifacts from LDC for 
an analysis of the quality of module design. We restricted our analysis to modules created 
during the 2016-2017 school year that included original uploaded student work samples, 
because these samples were required for module scoring. That restriction yielded a sample of 
115 modules that were authored or coauthored by 122 teachers (over half of all participating 
teachers) and three administrators. Given the presence of uploaded student work, these are 
modules that we are confident were implemented in the classroom. It should be noted, 
however, that as described in Chapter 4, almost 90% of teachers made at least one edit to a 
mini-task or module in CoreTools. The 115 modules are therefore part of a larger universe of 
modules worked on by participating teachers, some of which were likely implemented in the 
classroom despite the absence of uploaded student work. 


The school district required individually signed consent forms before releasing teacher 
data and teacher/student rosters, so for the outcomes analysis, we only received data on 
teachers who consented to participate in the study. The sample was further restricted by the 
need for student achievement data for both the outcome year (2016-2017) and the prior year 
(2015-2016). As a result, participants teaching either high school or lower elementary grades 
(K-3) could not be included in the student outcome analysis. These restrictions accounted for 
about half of the reduction in the sample from 217 consenting teachers to 118 teachers in the 
outcomes analysis. Specifically, 45 high school teachers and seven K—3 teachers were excluded. 


In addition, 10 teachers from one school in District 75, which serves special needs, were 
excluded. Another 37 teachers were excluded because they taught math or noncore content 
subjects, or were not associated with students with prior achievement and outcome year 
achievement scores in administrative data files. 


Table 2.2 
Number of Participating Schools and Teachers in NYC and Sample Size for Different Instruments 


Number of Number of schools in 
classroom Number of which teachers and 
teachers administrators administrators practice 
Participated in LDC 217 48 29 
Consented to CRESST study 196 36 29 
Completed survey 184 32 28 
Present in CoreTools dataset 202 33 29 
Author/coauthor of module with student 122 3 25 
work uploaded 
Included in outcomes analysis 118 NA 23 


2.3 Module Scoring Process 


LDC requirements intended that all teachers implement two modules over the course of 
the year, with the first spanning one week based on one text and the second spanning across 
multiple weeks and using multiple texts. Modules could be developed as original units of work 
or could be adapted from existing modules within the LDC CoreTools Library. Modules could 
also be either developed or adapted solo or collaboratively with other teachers within the PLC. 


Modules included in this analysis were collected from elementary and secondary teachers 
who participated during the 2016-2017 school year. As noted earlier, all modules were 
retrieved from LDC’s CoreTools online platform and examined during summer 2017. Artifacts 
that included both a completed module template and at least one student work sample, as 
determined by the evaluation team and/or expert raters, were then included in the analysis. As 
a result, 114 modules were rated and analyzed for teachers who participated in the i3 study. 
Additional modules that were missing a significant number of components or that were later 
found ineligible for the study were dropped after the rating sessions. 


Expert raters with experience teaching in the targeted grade spans and content areas 
were recruited from schools in Los Angeles County. The panel included two elementary school 
teachers, two secondary science teachers, two secondary social studies teachers, and three 
secondary English language arts (ELA) teachers. A greater number of raters was included for 
ELA since the number of modules eligible for analysis was greater for this content area. 
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Separate trainings were conducted for teachers from each grade span/content area. These 
trainings each lasted approximately one half-day, and were conducted by a member of the 
evaluation team with expertise on the Common Core and the rating of student and teacher 
artifacts. Each training session included an overview of the LDC goals, template task, the 
structure of the modules, and the CRESST rating dimensions. These dimensions were each rated 
using a 5-point scale and focused on (a) the effective writing task, (b) alignment to the CCSS and 
local and state literacy and content standards, (c) fidelity to LDC module instruction, (d) quality 
instructional strategies, (e) the coherence and clarity of the module, and (f) the raters’ overall 
impression (see Appendix A for full descriptions of the dimensions). 


Once the training was complete, calibration was conducted by having teachers 
individually score and then discuss their ratings for one module in the content area in which 
they were focusing. Scoring was then conducted on subsequent days with each module 
individually rated. All modules that were rated by at least two expert teachers were then 
discussed, with the goal, but not the requirement, to reach consensus (see Carlson & McCaslin, 
2003). All discussions were facilitated or supervised by the same member of the evaluation 
team who conducted the initial training. 


Because of the speed at which panelists were able to calibrate, while we initially planned 
for one third of the modules to be analyzed by more than one rater, the final rate of crossover 
was 55 or 48.2% of modules. It should also be noted that eight of these modules were rated by 
all three members of the Secondary ELA panel as part of their calibration activities. 


Table 2.3 presents a brief overview of modules rated. More specifically, 18% of the 
modules were at the elementary level with the remaining 82% of modules spanning middle 
school and/or high school. Both elementary and secondary modules were primarily focused on 
ELA. Most modules that were rated were regular modules, with only about one fifth being of 
the “one week, one text” format. Teachers at both grade spans were far more likely to adapt a 
module and most submitted their work with only one author listed. Elementary modules almost 
always included a student background section and, in addition, most included an extension 
and/or teacher reflection statement. In contrast, secondary teachers were more likely to 
provide a teacher reflection and were very unlikely to provide an extension. 
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Table 2.3 
New York Modules—Background Variables for all Modules Rated (N = 114) 


Elementary Secondary Total 
Variables # % # % # % 

Module type 

One week, one text 4 20.00 17 18.09 21 18.42 

Multiweek, multitext 16 80.00 77 81.91 93 81.58 
Origin 

Adapted 15 75.00 63 67.02 78 68.42 

Original 5 25.00 31 32.98 36 31.58 
Author count 

Coauthored 5 25.00 32 34.04 37 32.46 

Sole 15 75.00 62 65.96 77 67.54 
Subject 

ELA 14 70.00 61 64.89 75 65.79 

Science 3 15.00 16 17.02 19 16.67 

Social studies 3 15.00 13 13.83 16 14.04 

Other 0 0.00 4 4.26 4 3.51 
Module components 

Student background 19 95.00 50 53.19 69 60.53 

Extension 12 60.00 31 32.98 43 37.72 

Teacher reflection 14 70.00 60 63.83 74 64.91 
Total 20 17.54 94 82.46 114 100.00 


2.4 Survey Recruitment and Administration 


We began recruitment for the study by attending and soliciting consent forms at LDC 
Launch Days at the beginning of the 2016-2017 school year. The consent forms included 
language stating that the study was voluntary, all data would be protected, and that by signing 
the form, participants gave their consent to be emailed an electronic survey in spring 2017 and 
their permission for CRESST to request district data that linked the teacher to students. Directly 
compensating educators for participation in research is not permitted in New York state. We 
therefore offered instead a $300 gift card to the school to be used for school supplies to 
directly benefit students. The gift card was given to the school if the school’s project liaison and 
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at least five teachers participating in the PLC completed the survey. The $300 gift card 
represented an increase in the incentive from the pilot year, when schools were given a gift 
card equal to $20 per teacher or project liaison who completed the survey. The higher incentive 
award, as well as refinements to recruitment and survey completion protocols described below, 
raised the overall survey response rate to 81%, up from 52% the prior year. 


Most LDC participants were unable to attend the Launch Days at the beginning of the year 
so CRESST used other methods to recruit and collect consent forms. With LDC staff assistance, a 
CRESST staff member participated in a common planning session for each PLC either in person 
or via videoconference. For sessions joined via videoconference, consent forms were mailed to 
project liaisons in advance and project liaisons returned completed consent forms in a prepaid 
envelope or by scanning and emailing them back to CRESST. CRESST staff joined common 
planning sessions from October 2016 through June 2017. CRESST also followed up with 
individual participants to collect consent forms, but this method was less successful than 
recruitment during launch days and PLC sessions. 


Surveys were administered in late spring 2017. CRESST coordinated with LDC staff and 
coaches to administer the online surveys during common planning sessions. Multiple email 
reminders were sent to participants who were not present at the common planning session or 
who otherwise did not complete the survey. The teacher survey was closed at the end of the 
school year in June 2017. Administrator surveys were left open until early August when LDC 
staff met with administrators to plan for the 2017-2018 school year. 


2.5 Analytical Approaches 


Multiple analytic procedures, including both quantitative and qualitative analytic 
methodologies, were applied to the data to answer evaluation questions about how LDC was 
implemented, conditions affecting implementation, and program impacts. The following 
sections describe the methodological approaches used to analyze the collected data. 


Surveys. Descriptive statistics were generated for multiple-choice survey questions. Open- 
ended responses were reviewed for key themes and summarized. We collected responses from 
three groups of survey respondents: teachers, project liaisons, and administrators. Individuals 
who played more than one role are represented in the results for both roles. Therefore, the 
samples for the different analyses are not mutually exclusive. Generally, we report the number 
and percentage of respondents who selected different multiple-choice options. In a few cases 
where responses are numerical rather than categorical, we present means rather than 
proportions. 


Means and frequencies for all multiple-choice responses are presented in full in Appendix 
B for teachers, Appendix C for project liaisons, and Appendix D for administrators. Responses 
are organized by domain and specific questions are identified by question number. For 
example, the domain of LDC participation is summarized in one section, but relies on 
information from four questions. We preface teacher questions with “T,” project liaison with 
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“PL,” and administrator questions with “A”; please refer to the corresponding appendices for 
complete results on all items. 


LDC CoreTools. The first stage of our analysis examined the proportion of all participants 
who created CoreTools user accounts, and engaged in three key behaviors: viewing, editing, 
and commenting on modules. We then analyzed each of the three key measures of 
participants’ interaction with the LDC online tools, and reported means and/or frequencies as 
measures of participants’ engagement with the online LDC system. We calculated and 
presented total, average, maximum, minimum, and standard deviation for each of the 
measures in the dataset. In addition to reporting the overall results, we also provided results by 
the content subject the teacher taught (ELA, social studies/history, etc.), by participant role 
(project liaison or teacher), and school level (elementary, K-8, middle, 6—12, and high) 
whenever feasible. Finally, we explored the difference in CoreTools engagement between two 
groups of teachers: those that completed and taught modules and those that did not complete 
the design and implementation process. 


Modules. We used descriptive statistics (means, standard deviations) to analyze overall 
and subgroup performance for each content area/grade span on each of the six dimensions. 
Additionally, for the secondary ELA modules, which had a moderate sample size, we used 
generalizability theory (G theory) to examine potential sources of error during the rating 
process to help determine the validity of the scores as well as the construct validity of the 
rubrics (see Shavelson & Webb, 1991). Principal components factor analysis was also conducted 
to determine whether the six dimensions examined together represent one dominant or 
underlying trait for the secondary ELA sample. Finally, teacher comments during the debriefings 
for both elementary and secondary module ratings were examined to determine other 
potential issues with the rubrics and/or rating process. 


Generalizability theory is a statistical framework for examining multiple sources of 
potential error during the scoring process. For the secondary ELA sample, we first modeled 
score variability across all six dimensions using a two-faceted design, whereby we estimated 
variance components for module by rater by dimension (t x r x d). The goal here was to 
separate true variation in the quality of modules from other potential sources of measurement 
error. The main effects reflect true variation across modules (o2t) and error variance across 
raters (o2r) and dimensions (o2d), while the error term (o2trd,e) reflects unexplained residual 
error in the model. To disentangle the sources of potential error further, we also used a single- 
faceted design to examine potential error within the scoring of each dimension. As with the 
first set of models, the main effect reflects true variation across teachers (o2t) and error 
variance across raters (o2r). 


Student outcomes. We employed a quasi-experimental design to examine the effect of 
LDC on the New York State ELA assessment scores of students in the participating LDC 
elementary and middle schools in 2016-2017. Before conducting the analysis, we used a two- 
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step matching process to identify a reduced pool of comparison students and teachers at 
schools with similar characteristics to the schools in the intervention sample. 


To accomplish this, we first identified the five most similar control schools for each 
intervention school based on a Euclidian distance measure, by using the nearest neighbor 
analysis option in SPSS 24.0 (see Fix & Hodges, 1951; Wang, Neskovic, & Cooper). The variables 
used in this process were the percentage of students eligible for free or reduced price lunch, 
the percentage of African American students, mean prior student achievement in ELA, mean 
prior student achievement in math, the average attendance rate of teachers, the percentage of 
teachers with three or fewer years of teaching experience, and the school grade span where 
feasible. Once the pool of comparison schools was identified, their students and teachers were 
also identified and student-level matching was conducted so that the resulting sample would 
resemble the type of sample one would expect to obtain through random assignment. 


The student-level matching technique we employed was coarsened exact matching (CEM) 
(lacus, King & Porro, 2011). CEM is a flexible matching approach with many favorable 
properties, and allows the researcher to specify the precise conditions under which students 
are matched. For categorical variables, such as race/ethnicity or free or reduced price lunch 
status, this can entail exact matching, while for continuous measures, such as prior individual 
student achievement and aggregate class level achievement, cut-points for matching can be 
specified. With this approach we were able to set precise cut-points on the most important 
prior indicators, such as prior academic achievement, to ensure that where possible every 
treatment student was matched with a suitable comparison. Student matching variables we 
used in CEM included Hispanic, Black, poverty status, female, English language proficiency 
(English language learner), special education status, gifted status, mean prior achievement in 
math and ELA, and grade level. 


During matching we also included a few variables capturing information on the teachers 
and peers to which students were exposed. These variables included mean prior ELA 
achievement of the student’s peers in her core content classes, and the average years of 
teaching experience of student’s core content teachers. 


The typical structure of course taking at the middle school level involves students 
potentially being exposed to multiple teachers, with each responsible for a different core 
content class. Specifically, middle school students in the study had exposure opportunities 
across three content areas taught by intervention teachers (ELA, social studies/history, and 
science). As a result, students were not necessarily nested under individual teachers, but 
instead were likely to have connections to multiple teachers in the available time period prior 
to each testing outcome (students at the elementary school level were also sometimes exposed 
to multiple teachers but to a lesser extent). Therefore, LDC effects were estimated using an 
extension of the standard multilevel modeling framework known as multiple membership 
multiple classification (MMMC) models (Browne, Goldstein, & Rasbash, 2001). 
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These models can account for complex classification structures, such as the LDC context, 
in which students are nested within schools but are also members of multiple classes led by 
different teachers who may or may not be implementing LDC. MMMC has the flexibility to 
account for this type of complex nesting structure in which students are hierarchically nested 
under schools but may have one-to-many relationships with teachers. As can be seen in Figure 
2.1, there are three classification levels in the models we employ: students, teachers, and 
schools. In the MMMC modeling approach, each observation at the lowest level represents one 
student. The double arrows linking students to teachers in Figure 2.1 signifies the possibility of 
one student being exposed to multiple teachers. The single arrow from teachers to schools 
signifies that teachers were nested with schools. 


Students 


Figure 2.1. Multiple membership multiple classification structure. 


In the MMMC modeling approach, each observation at the lowest level represents one 
student. The weight each teacher receives with respect to each student is dependent on the 
student’s exposure to his or her teachers in each of the three core content areas. The total 
student exposure adds to a unity (i.e., a possible total exposure of 1) across their courses in the 
three content areas in a given school year. While this general weighting approach applies to 
both elementary and middle schools, the course structure of the datasets required us to use 
different weighting procedures in elementary and middle school. 


In both the elementary and middle school datasets, students were linked to teachers 
through statewide course identifiers and accompanying beginning and ending dates specifying 
the time each student was enrolled in a given course under a specified teacher. Students could 
potentially have data records connected to multiple teachers covering varying time periods in 
the same course. For each of the three core content courses we then collapsed the links into a 
single measure of the days of potential content exposure preceding the assessment period. The 
codes aligning with the three core content areas of interest for our study were based on the 
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SCED code handbook. In elementary school, in the event that a student was exposed to more 
than one teacher, each content area was given equal weight in distributing teacher/student 
exposure. For example, if a student was enrolled for both ELA and social studies/history under 
one teacher, then that teacher was coded as .67 for having contributed to two thirds of the 
students’ core curriculum exposure. If the same student enrolled in science with a different 
teacher than the one who was linked to their course marks in ELA and social studies/history, 
then that science teacher would have been coded as .33 and all other teachers in the sample 
would have been coded as zero. This would then result in the student’s exposure adding to a 
unity (1). 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead each core content 
area exposure contributed to a core content area total sum which formed the basis from which 
the weights were proportioned. Most commonly a student had equivalent days of core 
instruction exposure in each of the three content areas (often 214 days in each content area). 
In that scenario, if a student had exposure to three different teachers, then each teacher would 
contribute one third (.33) of the overall core curriculum exposure and all other teachers in the 
sample would be coded as zero. However, in addition to the typical core science course, extra 
core science courses were also included in the LDC analysis (for example a Grade 8 student 
taking biology), which made it possible then for a student to accumulate more units in science 
than in the other two content areas. The weighting in middle school was always distributed as a 
proportion of the total exposure days in the three content areas. Therefore, if a student 
accumulated 300 science days (across two courses), 200 social studies days, and 200 ELA days, 
the base number of instruction days would be 700 days. If, using that same scenario, the same 
teacher taught both the typical core and biology courses then that teacher would contribute 
three-sevenths (.43) of the overall core curriculum exposure with the social studies and science 
teachers contributing two-sevenths (.285) each, again resulting in the student’s exposure 
adding to a unity (1). Tabular versions of the above examples can be seen in Appendix E. 


For this study, we modeled the treatment intervention variable as a fixed effect at the 
student level in two different ways. The first dosage-dependent approach takes into account 
the students’ level of exposure to the intervention teachers. In this approach, the treatment 
was structured as a continuous response variable, coded as zero for comparison students and 
coded as a positive value for treated students, albeit never exceeding one. The positive value 
assigned to treated students in the dosage-dependent approach was simply the sum of the 
intervention teacher weights linked to the treated student. The second approach was dosage 
independent and classified any student exposed to an intervention teacher via at least one 
course as a treated individual. In this approach the treatment variable was dichotomous (coded 
as one for treated students and zero for comparison students). 
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As with other multilevel models, MMMC models account for the nonindependence of 
observations within cluster by adjusting the inferences on parameter estimates for the 
correlations between responses in a cluster. This modeling approach, however, becomes 
computationally cumbersome using traditional frequentist estimation methods. As 
recommended by Browne et al. (2001) we instead employed Bayesian methods using Monte 
Carlo Markov chain (MCMC) techniques to best address this issue. Multilevel models 
incorporate demographic and achievement variables used in the matching design as covariates, 
making the findings “double robust” (characteristics controlled for in both matching and 
outcomes analysis stages). Student demographic and prior achievement variables that were 
used in the matching process were also included as covariates in the MMMC model. In the 
elementary analysis, variables for ELL students, students with disabilities, and gifted students 
were not significant contributors to the models so they were not included in the final 
elementary models. The full specifications for both models can be found in Appendix E. 
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3.0 Survey Analysis 


Five groups of participants were surveyed: (1) teacher, (2) teacher/project liaison, (3) 
project liaison, (4) administrator, and (5) administrator/project liaison. Twenty-nine schools 
began implementing LDC in the 2016-2017 school year, with 217 classroom teachers 
participating and 48 administrators overseeing the work (see Table 3.1). The 29 NYC schools 
included two elementary, four K-8, 17 middle, three 6-12, and three high schools. Two of the 
schools had two PLCs, while the remaining 27 schools had one PLC. Participants taught across 
all grade levels from K to 12. Most secondary teachers taught ELA, social studies/history, or 
science, with a smaller number teaching math. 


Table 3.1 
Survey Consent and Response Rates: 2016-2017 


Number of | Number of Number of Consent Response 
Participant type participants consents survey responses rate rate 
Teacher 188 168 159 89% 85% 
Teacher/project liaison 29 28 25 93% 83% 
Total teachers 217 196 184 90% 85% 
Project liaison 1 1 0 100% 0% 
Administrator 42 33 29 79% 69% 
Administrator/project liaison 4 1 3 25% 75% 
Total administrators? 48 36 32 15% 67% 
Total participants 265 232 216 88% 82% 


We categorize the coaches and coordinators who completed the project liaison survey as administrators. 


Table 3.1 presents the study consent and survey response rate information. As shown, 
survey consent and response rates were different for teachers who implemented LDC in their 
classrooms and administrators. Compared to administrators, teachers had a considerably 
higher consent rate (90% compared to 75%) and survey response rate (85% compared to 67%). 
Teachers who were project liaisons were more likely to consent to participate in the study but 
no more likely to complete the survey. Overall, consent and response rates were quite high and 
were markedly improved from the pilot year of the study. 


We discuss survey results according to the domains listed in Table 2.1. We also include 
specific questions covered in each domain for easier reference. For example, LDC Participation 
(T1-4) indicates that teacher questions 1-4 are used to provide information on LDC 
participation. Multiple choice survey questions and descriptive results (frequencies and means) 
are presented in full in Appendix B for teachers, Appendix C for project liaisons, and Appendix D 
for administrators. As noted earlier, these three samples are not completely mutually exclusive 
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(i.e., some teachers and administrators also acted as project liaisons). Results are organized by 
domains and question numbers. For example, the domain of LDC participation is summarized in 
one section, and relies on information from four different questions. We preface teacher 
questions with “T,” project liaison with “PL,” and administrator questions with “A.” 


Survey results are presented here in four sections. The first section summarizes teachers’ 
responses. The second section summarizes project liaisons’ responses, and the third section 
summarizes administrators’ responses. Whenever we felt a comment from an open-ended 
response might clarify, illustrate, or corroborate a finding, we included that comment in the 
appropriate section. The fourth section summarizes open-ended responses from all 
participants, who answered the same three questions about program efficacy and 
improvement. We end with a summary of results. 


3.1 Teacher Survey Results 


A total of 184 teachers at 28 schools completed the survey (see Table 3.2). One 
participating middle school did not return any surveys. Of the 184 teachers who responded to 
this survey, 5% taught in two elementary schools, 23% in four K—8 schools, 54% in 16 middle 
schools, 13% in three 6-12 schools, and 18% in three high schools. Twelve teachers (7.5%) 
indicated they had not participated in a professional learning community in the 2016-2017 
school year. Therefore, they did not receive most of the survey questions. 


In addition to producing descriptive statistics on the whole teacher sample, we also 
produced results for elementary and secondary level teachers. We highlight important 
differences between the two subgroups, when they are apparent. Note, however, that the 
elementary sample included just 19 teachers, and therefore comparisons should be interpreted 
with caution. 


Table 3.2 
Schools and Teachers Completing the Survey in 2016-2017 


School type Number of schools Number of teachers % of teachers 
Elementary schools 2 10 5% 
K-8 schools 4 23 13% 
Middle schools 16 95 52% 
6-12 schools 3 23 13% 
High schools 3 33 18% 
Total 28 184 100% 


LDC participation (T1—4). Almost all teachers (90%) reported that this year was their first 
experience with LDC in questions T1-4. The remaining 19 teachers who had prior experience 
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with LDC reported that they had taught 0 to 6 modules (M = 1.2) or 0 to 8 mini-tasks (M = 1.2) 
outside of modules. Nine of these 19 teachers taught zero modules, and 12 of these teachers 
taught zero mini-tasks. 


Among the 184 teachers who completed the survey, 15 of them taught in elementary 
grades, and 169 taught secondary grades. Secondary teachers reported teaching 1 to 15 classes 
(M = 3.7); in these classes, they used LDC materials in 0 to 5 classes (M = 2.4). Five out of the 
166 teachers who provided information about their content area implemented modules in two 
or three content areas, resulting in 172 content area implementations. “Humanities” includes 
teachers who taught both ELA and history/social studies. ELA includes teachers who taught 
English as a New Language (see Figure 3.1). 


Content Area Implementations 
50% 
45% 
40% 
35% 
30% 
25% 
20% 


15% zi mak 10% 
10% 
0% (| 


ELA (n=75) = Science (n=33) History/Social © Humanities Math (n=18) Other (music, 
Studies (n=23) (n=20) Spanish) (n=3) 


44% 


19% 


Figure 3.1. Secondary teachers’ content area implementations. 


Professional learning community and teacher collaboration (T5—9, T30, T37). Almost all 
teachers (93%) participated in a PLC that was at least partly focused on implementing LDC. Of 
the 12 teachers who had not participated in a PLC, nine teachers (75%) reported they had used 
LDC tools in their planning or instruction, primarily CoreTools to access existing modules or to 
design modules. 


Almost two thirds (65%) of teachers reported meeting in their LDC PLC at least once a 
week or more. About a quarter (26%) met every other week, and 9% met once a month. The 
most common reason cited for not meeting weekly was that PLC members had other priorities 
(57%). Interestingly, secondary teachers on average met in PLC meetings less frequently than 
elementary teachers. About 10% of secondary teachers reported meeting once a month or less 
frequently while no elementary teachers reported such infrequent meetings. Likewise, all 
elementary teachers met weekly or more as recommended compared to about 60% of 
secondary teachers. 
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According to teachers, LDC PLC meetings generally lasted 45 minutes to an hour (54%). 
Almost a third of teachers (30%) reported that they lasted longer than an hour, and 17% less 
than 45 minutes. Most teachers (74%) agreed or strongly agreed that their PLCs were given 
sufficient time to meet. Secondary teachers were more likely to report meeting for an hour or 
more (about a third did so) than elementary teachers (7%). 


Almost half (49%) of teachers said they had informal discussions about LDC with their 
colleagues once a week or more; 22% every other week; 11% once a month; and 19% less than 
once a month. As with formal meetings, secondary teachers met less frequently than 
elementary teachers; for example, two thirds of elementary teachers met informally twice a 
week or more compared to just 17% of secondary teachers. Over two thirds (70%) of teachers 
agreed or strongly agreed that they were more likely to collaborate with other teachers on 
designing instruction after participating in LDC, and 72% reported that LDC participation helped 
them develop working relationships with teachers in different grades or subjects. 


LDC training and support (T10-T13). Teachers evaluated the three main types of LDC 
training and support: professional learning community (PLC), online course materials, and 
virtual coach support during and outside of PLCs. 


Teachers found PLCs to be moderately effective to very effective in the following ways: 
creating an environment in which teachers were comfortable working with each other (79%); 
allowing space for shared problem solving (77%); helping teachers learn to develop modules 
(75%); fostering an environment where teachers shared instructional plans with colleagues 
(74%); and demonstrating the usefulness of LDC (65%) (see Figure 3.2). Elementary teachers 
were particularly positive in their responses, with 100% reporting that their PLC was effective in 
each way. 
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Figure 3.2. The effectiveness of teacher’s PLC (T10). 


All aspects of the online course materials were rated by a majority of teachers as good or 
excellent (see Table 3.3): clarity of information presented (68%); relevance of information 
(68%); usefulness of resource documents such as the LDC Pitfall Checklist (62%); helpfulness in 
creating modules (62%); opportunity to extend learning (62%); ease of use (61%); and 
usefulness of videos (57%). Again, elementary teachers were even more positive, with 100% 
rating all of these features as good or excellent. 
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Table 3.3 
Rating Online Course Materials (n = 170) 


How would you rate each of the following aspects of the online 
course material (in the Learn tab in LDC CoreTools) that your coach 


used or directed you to use? Poor Fair Good Excellent 
Clarity of information presented 7% 26% 47% 21% 
Relevance of information presented 9% 23% 41% 27% 
Ease of use 13% 27% 44% 17% 
Usefulness of resource documents (e.g., LDC Pitfall Checklist, CCSS 8% 31% 43% 19% 
Mental Markers, etc.) 
Usefulness of videos 10% 33% 45% 12% 


Degree to which course material helped me to create and/or adapt 11% 27% 45% 17% 
LDC modules 


Opportunity to extend learning when needed or desired 11% 28% 38% 24% 


Almost all teachers (95%) said they were able to get the feedback and support they 
needed from their LDC coach and that coaches provided written feedback in a timely manner 
(95%). The following types of coach support were found to be moderately or very helpful: 
written feedback in CoreTools (69%; 8% did not use); individual Zoom conference (61%; 15% 
did not use); facilitated discussion on Teaching Channel Teams (54%; 23% did not use); and 
email or phone communication (53%; 29% did not use). 


LDC implementation (T14—28, T30). This domain covers questions on module creation 
(T14-17), module peer review (T27—28), and classroom implementation (T18—26, T30). 


Module creation (T14—17) and module peer review (T27-28). Teachers adapted or 
created two types of LDC instructional products: mini-tasks are short, generally taking one 
period, and focus on a specific skill; modules are longer, more complex units composed of 
multiple mini-tasks, ending in a culminating “teaching task.” 


Using existing LDC templates and exemplars, among the 145 teachers who answered this 
question, 25 teachers (17%) adapted zero modules, 109 teachers (75%) individually or 
collaboratively adapted between one and three modules during the year, 10 teachers (7%) 
adapted four to seven modules, and one teacher (1%) reported adapting 20 modules. The mean 
number of modules teachers adopted is 1.9 modules. When asked about creating modules from 
scratch, 30 teachers (21%) reported zero modules, 110 teachers (76%) individually or 
collaboratively constructed one to three modules, and five teachers reported (3%) either four 
or five modules. Among these 145 teachers, the mean number of modules they created is 1.6 


24 


modules. Elementary teachers chose to adapt modules more often than secondary teachers, 
while secondary teachers were more likely to create new modules from scratch. 


Majorities of teachers reported creating modules individually (61%) and as part of ateam 
(63%) (the choices were nonexclusive). A minority of teachers (11%) reported creating modules 
with the PLC as a whole (this collaboration structure was more common among elementary 
teachers with over a third reporting that modules were created in this way). Among the 170 
teachers who answered the question on submitting modules for LDC National Peer Review, 88 
teachers (52%) did not submit any modules for LDC National Peer Review, 71 teachers (42%) 
reported submitting one or two modules for review, seven teachers (4%) reported submitting 
three modules, and the remaining four teachers (2%) submitted either four or five modules. Of 
the 82 teachers who reported submitting modules for peer review, nine (11%) found the 
process very helpful, 30 (37%) moderately helpful, 26 (32%) a little helpful, and 17 (21%) not 
helpful. 


Teachers were generally confident in their ability to create effective LDC modules and 
mini-tasks: 85% felt they had been able to select a set of focus standards for a writing task to a 
moderate or great extent; 84% felt they had been able to create a standards-driven writing 
assignment task; 82% said they had identified the skills students need to develop in order to 
complete the writing assignment; 79% reported they had created a writing assignment that 
allowed students to engage with the material; 75% said they selected high-quality texts and 
other materials to encourage deeper learning; 71% made connections to previous or future 
learning to make the task relevant; 69% created daily lessons to teach skills needed to complete 
the writing assignment; and 65% were able to plan a variety of methods to assess student 
progress. Please see Table 3.4 below for more detailed information. Elementary teachers were 
particularly confident with 90 to 100% reporting that they were able to accomplish each of the 
tasks listed in Table 3.4. 
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Table 3.4 
Rating on Teacher Skills Associated with Creating LDC Modules and/or Mini-Tasks (n = 170) 


Please indicate to what extent you were able to do each of 


the following when creating LDC modules and/or mini- Amoderate A great 
tasks. Notatall Alittle bit | extent extent 
Select a set of focus standards for a writing assignment 1% 14% 35% 51% 
Create a standards-driven writing assignment task 2% 15% 31% 53% 
Select high-quality, complex texts and other materials that 6% 19% 39% 37% 


allowed students to engage in deeper learning 


Create a writing assignment that provided multiple 5% 16% 42% 38% 
opportunities for students to engage with the material 


Identify the skills students need to develop in order to 4% 14% 42% 40% 
complete a writing assignment 


Create daily lessons to teach the skills a student needs to 8% 23% 41% 28% 
complete a writing assignment 


Plan for a variety of methods to assess student progress 7% 28% 42% 24% 
(e.g., mini-task scoring guides) 


Make connections to previous or future learning that 8% 21% 41% 31% 
make a writing assignment relevant for students 


Classroom implementation (T18-26). After creating their modules, teachers implemented 
modules in their classrooms and reflected on their progress. Teachers implemented 0 to 8 
modules (M = 2.4) and implemented from 0 to 35 mini-tasks (M = 3.6) that were not part of 
modules. Table 3.5 presents the number of modules and mini-tasks teachers implemented per 
their reporting. The majority of the 170 teachers, 51% or 86 teachers, reported implementing 
two modules in their classrooms, and 34% or 57 teachers reported implementing three or more 
modules in their classrooms. Meanwhile, 22% or 37 teachers reported implementing zero mini- 
tasks outside of modules. 
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Table 3.5 
Number of Modules and Mini-Tasks Implemented (n = 170) 


Numbers of modules or mini-tasks Number of teachers Number of teachers 
implemented in 2016-2017 implementing modules implementing mini-tasks 
0 3 37 
1 24 15 
2 86 37 
3 38 21 
4 14 13 
5 11 
6 2 8 
7 1 3 
8 2 4 
9 3 
10 13 
11 1 
12 1 
15 1 
16 1 
35 1 


Teachers reflected on their ability to provide instruction through the modules. Across all 
six dimensions, the distribution of teachers who felt they were able to implement effective 
instructional strategies through the modules was high: 85% felt they had engaged students in 
productive struggle to a moderate or great extent; 79% differentiated instruction; 77% located 
performance of standards in student work; 76% used evidence of learning to modify 
instruction; 76% provided feedback to students using shared expectations of learning; and 73% 
systematically collected information about student learning. Also across all six dimensions, the 
percentage of teachers who felt they had accomplished these strategies to a great extent 
averaged 32% (range 27-34%), indicating that about a third of teachers felt very confident 
about their ability to implement high-quality instructional strategies using LDC modules. Again, 
elementary teachers exhibited a high degree of confidence, with 90 to 100% reporting they 
were able to implement each of the instructional strategies listed above. 
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The One-Text One-Week module was introduced as a first step toward module creation. 
As its name indicates, the module highlighted one text that was examined closely over one 
week. About half of teachers (51%) taught a One-Text One-Week module. Of those teachers, 
72% created the module from an LDC template, while the remaining 28% copied and adapted 
another teacher’s module from the LDC online library. 


Following the One-Text One-Week modules, teachers typically worked on a longer term 
instructional module. About a third of teachers (34%) copied and adapted another teacher’s 
module from the LDC online library, while 67% created a module from an LDC template. Most 
teachers (82%) taught this module during the school year, while 8% planned to teach it before 
the current school year ended, 6% planned to teach it the following school year, and 4% had no 
plans to teach the module. 


Teachers submitted 0 to 5 modules (MV = 0.8) for LDC National Peer Review. Of the 82 
teachers who reported submitting modules for peer review, nine (11%) found the process very 
helpful, 30 (37%) moderately helpful, 26 (32%) a little helpful, and 17 (21%) not helpful. 


Leadership support (T32—36). This domain covers questions on project liaison support 
and administrator support (T32—33) and teacher leadership role in LDC (T36). 


Project liaison support (T32). Project liaisons were school staff (participating LDC 
teachers, school administrators, or coaches) who provided logistical support to the PLCs. 
Almost all teachers were very satisfied with the level of project liaison support. The project 
liaisons were approachable (93% agreed or strongly agreed); effectively supported the PLC 
meetings (92%); effectively invited teachers to join LDC (90%); had a strong grasp of LDC (88%); 
gave useful feedback (86%); and helped teachers align LDC to broader school goals (85%). 


School administrator support (T33—35). School administrators were principals, assistant 
principals, or other instructional leaders who observed teachers in action and provided other 
leadership, such as protecting time for PLCs to meet. In questions T33—35, we asked teachers to 
provide feedback about the support they received from their school administrator. Almost all 
teachers (93%) agreed or strongly agreed that their administrator encouraged LDC 
participation. According to teachers, most administrators were able to ensure PLCs had time to 
meet (86%); communicate how LDC supported school initiatives and goals (76%); and had a 
firm understanding of LDC (72%). Most administrators also made formative assessment a 
priority (88%) as reported by teachers. Over half of teachers reported that their administrator 
had provided feedback about LDC planning and instruction (58%). A minority of teachers 
reported that administrators expressed concern that LDC was taking time away from other 
instructional priorities (42%). 


The involvement of school administrators was varied. About a quarter (26%) of teachers 
reported their administrators attended three quarters or more of their PLC meetings. On the 
other hand, 42% reported their administrators attended less than one quarter of meetings. The 
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remaining teachers reported that administrators attended one quarter (18%) or one half (15%) 
of meetings. Administrator attendance was on average higher in elementary school PLCs. A 
large minority of teachers (44%) reported never being observed by an administrator while 
teaching an LDC mini-task. About a third (32%) reported being observed once; 12% were 
observed twice; and the remaining 13% were observed three or more times. Again, on average 
elementary teachers reported being observed a greater number of times than secondary 
teachers. 


Teacher leadership role (T36). The role of teachers in LDC extends beyond the classroom. 
As leaders in their schools, LDC teachers are pivotal in driving LDC implementation. Most 
teachers agreed or strongly agreed that LDC helped them create writing assignments within 
their current curriculum (78%). Substantial proportions of teachers felt they were able to affect 
the direction of LDC at their site by having the opportunity to work with the project liaison and 
administrator (59%); setting instructional goals for LDC work (62%); and being involved in 
discussions about expanding LDC implementation in future years (48%). Almost two thirds of 
teachers (65%) expressed interest in learning more about facilitating LDC implementation at 
their schools. Elementary teachers were much more likely to report feeling empowered by the 
LDC program; 90 to 100% agreed that they were able to play a leadership role in each of the 
ways listed above. 


LDC impact (T29-31). This domain covers questions on LDC impact on teacher practice 
and learning (T29-30) and student learning (T31). 


Impact on teacher practice and learning (T29-30). We asked teachers about LDC’s impact 
ona number of teacher skills. Specifically, we asked them to focus on the change between the 
beginning and end of the current school year’s work with LDC. We also asked them to provide 
more information about the impact of LDC on their instructional practice, including their use of 
LDC tools and their level of collaboration with colleagues. 


Over all eight items about change in ability during the year, about two thirds of teachers 
rated themselves as having improved moderately or a great deal. Specifically, the areas for 
which the greatest number of teachers reported a moderate or great impact involved 
instructional planning: identifying skills needed to complete an assignment (67%); selecting 
focus standards (69%); and creating high-quality writing tasks (70%). The other areas of impact 
on teacher practice and learning that received slightly lower ratings concerned collecting and 
using student data to inform instruction: assessing student progress (65%); using evidence of 
student learning (66%); identifying student misconceptions (63%); and tracking and analyzing 
student progress (62%). Elementary teachers were more likely to report change in their abilities 
than secondary teachers, with over three quarters reporting moderate or great change on each 
of the eight items. 
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Figure 3.3. Impact on teacher practice and learning (T30). 


The areas of impact on teacher practice most commonly reported by teachers (see Figure 
3.3) included development of relationships with teachers outside grade or subject (72% agreed 
or strongly agreed); incorporation of writing assignments into existing curricula (70%); likeliness 
of collaboration with other teachers (70%); and raising expectations for student writing (70%). 
LDC also had slightly less but still apparent impact in the following areas: incorporation of state 
standards into instruction (66%); use of LDC modules as a part of existing instructional practice 
(62%); and incorporation of LDC mini-tasks into non-LDC instruction (57%). Again, elementary 
teachers reported positive impacts on teacher practice more frequently than secondary 
teachers. 


Impact on student learning (T31). Teachers were asked to evaluate the effect of LDC on 
student learning (see Table 3.6). The majority of teachers agreed or strongly agreed that LDC 
supported student learning. The two areas for which the greatest number of teachers reported 
impact involved writing: supporting students to complete writing assignments (74%) and 
increasing student capacity to analyze components of the writing task (74%). According to 
teachers, LDC supported skills needed for college and career readiness (72%) and overall helped 
literacy performance (70%). LDC also reportedly improved quality of writing (69%), reading 
skills (69%), and content knowledge (69%). Finally, LDC helped students develop speaking and 
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listening skills (65%). Again, elementary teachers were particularly confident in the impact of 
LDC on student learning, with over 90% reporting a positive impact on nearly every skill area. 


Table 3.6 
Impact on Student Learning (n = 144) 


Please indicate the degree to which you agree or Strongly Strongly 
disagree with the following statements. disagree Disagree Agree agree 

LDC helped students develop reading skills. 10% 21% 56% 13% 
LDC was effective in improving students’ content 10% 22% 53% 16% 
knowledge. 
LDC modules effectively supported students in 8% 18% 57% 17% 
completing writing assignments. 
LDC was effective in improving the quality of students’ 10% 22% 51% 17% 
writing. 
LDC supported students’ development of skills needed 10% 17% 57% 15% 
for college and career readiness. 
LDC increased students’ capacity to analyze and 8% 18% 54% 20% 
understand the components of a writing assignment 
task. 
LDC helped students develop speaking and listening 10% 26% 51% 14% 
skills. 
Overall, LDC helped improve students’ literacy 10% 21% 54% 16% 


performance. 


Facilitators and barriers (T37). Successful implementation of LDC depends on a number of 
factors. We asked teachers to weigh in on the effect of these factors on implementation (see 
Figure 3.4). Over three quarters (79%) agreed or strongly agreed that their school had adequate 
technology to support teachers’ use of LDC. Most teachers (74%) felt their PLC was given 
sufficient time to meet. Teachers were able to easily find and adapt LDC mini-tasks indicating 
that they were able to use the CoreTools online platform successfully (71%). Most teachers 
(71%) felt adequately prepared to implement modules in the classroom. About half of teachers 
(52%) found it challenging to find content-rich texts to use with LDC modules. 
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Figure 3.4. Facilitators and barriers (T37). 


3.2 Project Liaison Survey Results 


At each school, the project liaison coordinated PLC meetings with LDC coaches. Project 
liaisons were participating teachers, school administrators, or other school support staff. A total 
of 30 project liaisons responded to the survey. 


LDC participation (PL1—2). Of the 30 project liaisons who responded, 26 (87%) reported 
that they did not have prior experience with LDC. Of the four who had prior experience, one 
reported having taught six modules, two reported teaching two modules, and one reported 
having taught no modules. 


Professional learning community and teacher collaboration (PL3-6). Nearly three 
quarters (73%) of project liaisons reported that their PLCs met once a week or more frequently. 
Of the eight project liaisons whose PLCs did not meet weekly, three said PLC members had 
competing priorities and two reported that the primary barrier was that time was not 
protected. Other reasons given were that coach-led meetings were biweekly; it was difficult to 
schedule common planning time because it was a small school; and it was difficult to schedule 
time because the school was closing. 


Over a third of project liaisons (37%) said these meetings lasted 45 to 59 minutes, which 
corresponds with the available planning period of 45 minutes. A quarter (27%) said meetings 
were less than 45 minutes, 27% said they were 60—74 minutes, and 10% reported longer than 
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75 minutes. Almost half (47%) reported that they had informal discussions about LDC with 
teachers once a week or more. 


LDC training and support (PL7—12). Project liaisons evaluated the effectiveness of the 
various training and support methods provided during LDC implementation. These included PLC 
meetings, online course material, and coach support. 


Project liaisons overwhelmingly reported that the PLC meetings were effective. Nearly all 
(94%) felt PLC meetings were moderately or very effective in creating an environment in which 
teachers were comfortable working with each other; 90% said the PLC allowed space for shared 
problem solving; 90% thought the meetings helped teachers learn to develop modules, 84% felt 
they fostered an environment where teachers shared instructional plans with colleagues; and 
67% thought they demonstrated the usefulness of LDC to teachers. 


Online course materials used by the coaches during PLC meetings also received high 
ratings from the project liaisons. Most project liaisons (87%) rated the relevance of information 
as good or excellent; 87% for clarity of information; 80% for helpfulness to teachers in creating 
modules; 73% for usefulness of resource documents; 73% for ease of use; 73% for usefulness of 
videos; and 73% for opportunity to extend learning when needed or desired. 


LDC coaches were highly regarded by project liaisons. Nearly all (97%) reported that they 
were able to get the feedback and support they needed from LDC coaches, and that the 
coaches provided written feedback on modules in a timely manner. Other types of coach 
support were rated by the majority of project liaisons as moderately or very helpful. These 
supports were written feedback in CoreTools (87% helpful; 3% did not use); email or phone 
communication (83% helpful; 3% did not use); individual video conferencing with coaches (80% 
helpful; 3% did not use); and coach-facilitated discussions on Teaching Channel (67% helpful; 
7% did not use). All project liaisons (100%) agreed or strongly agreed that their LDC coach was 
knowledgeable and provided high-quality guidance; 97% thought their coach was easy to work 
with; 97% said their coach responded quickly; 93% reported being able to reach their coach 
with questions; 87% said coaches connected them with additional resources when needed; and 
only 30% of project liaisons reported that it was challenging to structure PLC time with the 
coaches. 


Project liaisons had the opportunity to attend LDC professional development meetings, 
both in person and online. The average number of meetings attended was 3.4, with a range of 0 
to 15. The majority (55%) participated in two or three meetings; 15% attended no meetings. 


Finally, project liaisons provided feedback about non-coach LDC support. Almost all (90%) 
project liaisons believed LDC offered sufficient professional learning opportunities to help them 
lead the initiative at their school, and 80% agreed that LDC provided adequate technical 
support for issues with the CoreTools online platform. 
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LDC implementation (PL 13-16, PL20). This domain covers questions on module creation 
(PL13-16) and alignment (PL20). 


Module creation (PL13—16). Project liaisons reported the ways in which modules were 
created in their PLCs and judged the extent to which they were able to embed targeted 
instructional practices while creating the modules. 


Project liaisons were asked how many LDC modules were created by their PLC members, 
either individually or collaboratively by adapting them from existing modules created in a prior 
year or found in the LDC Library in CoreTools. Answers ranged from 0 to 4, with a mean of 1.2. 


In general, project liaisons were extremely confident in their ability to display evidence of 
the targeted instructional practices while creating their modules. All project liaisons (100%) felt 
they were able, to a moderate or great extent, to select focus standards for the writing 
assignment and identify skills necessary for students to complete the assignment. Nearly all 
(97%) felt they were able to create a standards-based writing assignment task; 90% provided 
multiple opportunities for students to engage with the assignment; 86% were able to select 
high-quality texts and other materials; 86% made connections to previous or future learning to 
make the assignment relevant for students; 76% created daily lessons to teach the necessary 
skills; and 72% planned for a variety of ways to assess student progress. 


Alignment (PL20). Project liaisons were asked how LDC aligned with other instructional 
priorities and programs at their schools. Alignment was generally perceived to be quite high as 
reported in Figure 3.5. Nearly all (93%) agreed or strongly agreed that LDC helped teachers 
create writing assignments to use within their existing curricula; 86% believed LDC 
complemented other initiatives at the school; 86% viewed LDC as a strategy for implementing 
statewide college and career-ready standards; 79% thought LDC helped prepare students for 
state assessments; 79% reported their school connected LDC to specific school goals; and only 
41% reported that time spent on LDC interfered with other initiatives. Notably, however, 
almost three quarters (72%) believed that it was difficult for teachers to focus on LDC because 
of other competing priorities at the school. 
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Figure 3.5. LDC alignment (PL20). 


Leadership support (PL17—19, PL22). This domain covers questions on school 
administrator support (PL17—18), project liaison leadership role (PL19), and district support 
(PL22). 


School administrator support (PL17—-18). Project liaisons reported on school 
administrators’ involvement with LDC. Almost half (48%) reported their administrators 
attended less than a quarter of PLC meetings; 15% about a quarter; 19% about half; 7% about 
three quarters; and 11% more than three quarters. 


Almost all (97%) reported their administrators made formative assessment a priority at 
their school; 93% said they received feedback about their LDC leadership; 93% reported their 
administrators encouraged teachers to participate; 89% felt planning time for LDC was 
protected; 85% said their administrator communicated how LDC supported other school goals 
and initiatives; 78% of school administrators were said to have a firm understanding of LDC; 
and only 41% of project liaisons reported that their administrators expressed concern that LDC 
was taking time away from other instructional priorities. 


Project liaison leadership role (PL19). Project liaisons reflected on their leadership role in 
LDC implementation at their school sites. Overall, results were very positive and indicated that 
the majority of project liaisons felt engaged by and invested in the LDC implementation at their 
school. Almost all (93%) agreed or strongly agreed that they met regularly with their LDC coach 


to manage the work plan; 67% met regularly with their school administrator; 85% felt they 
were able to build capacity as an instructional leader as a result of their LDC leadership role; 
78% were involved in planning how to meet teacher learning needs by differentiating LDC 
implementation; 78% discussed how to expand LDC implementation in future years; 78% 
helped set instructional goals for LDC implementation; and 59% believed their role as LDC 
project liaison helped them advocate for additional resources on their campus. 


District support (PL22). By and large, project liaisons were unaware of the nature and 
extent of district support for LDC. Typically about half of project liaisons reported “don’t know’ 
in response to the following statements about district participation and support, while, on 
average, about a third agreed or strongly agreed: district leaders supported LDC 
implementation (38%); district leaders had a firm understanding of LDC (31%); district leaders 
were interested in spreading LDC to more schools (28%); district professional development 
efforts aligned with LDC (31%); and district leaders visited schools to discuss LDC (38%). 
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Scale-up and sustainability (PL21). Regarding the future of LDC implementation, most 
project liaisons (83%) agreed or strongly agreed that they expected most teachers to continue 
with LDC the following year; 76% felt teachers and administrators were committed to 
sustaining the initiative; and 62% expected their LDC PLC to increase in size the following year 
(see Figure 3.6). Generally, however, they did not believe LDC had yet reached beyond the PLC, 
with just 21% reporting that non-LDC teachers used LDC resources, such as the planning 
process or CoreTools. 


0, 9 
70% 62% 62% 
60% 
a0 38% 38% 
40% 
30% 17% 21% ane 
20% 
~ io i | | 
0% 
| expect that most Teachers at my Asaresult of LDC, Teachers and | expect our LDC 
teachers school who were new administrators at PLC to increase in 
participating in not part of the collaborations my school are size next year. 
LDC this year will LDC PLC meetings across grades committed to (n=29) 
continue to do so used the LDC and/or subjects sustaining the 
next year. (n=29) planning process were created or LDC initiative. 
and/or LDC are being (n=29) 
CoreTools. (n=29) launched at my 
school. (n=29) 


@ Agree/Strongly agree O Disagree/Strongly Disagree 


Figure 3.6. Scale-up and sustainability (PL21). 
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3.3 Administrator Survey Results 


This section presents the survey results from school administrators. 


LDC participation (A1) and professional learning community (A2). Of the 32 school 
administrators who responded to this survey, 18 (56%) identified as principals, 13 (41%) 
identified as assistant principals, and one (3%) identified as the LDC liaison. In some schools, 
more than one administrator was involved in overseeing the LDC work. 


Administrators reported that they attended relatively few sessions of LDC PLCs, with 44% 
attending less than one quarter, 19% attending about one quarter, and 13% attending about a 
half. Another 13% reported attending three quarters, and 13% attended more than three 
quarters. These numbers align with teachers’ reports of their administrators’ attendance at 
PLCs. 


Training and support (A3—4). Administrators had the opportunity to attend professional 
development sessions, either online or in person, such as Launch Day and administrator 
meetings. Two thirds of administrators attended one (40%) or two (27%) of these meetings. 
The average was 2.7. 


Interactions with LDC were overwhelmingly positive. Almost all administrators agreed or 
strongly agreed that LDC offered sufficient professional development for project liaisons (97%); 
their school had adequate technology to access LDC (97%); they were able to reach LDC staff 
with questions (93%); there were sufficient professional development opportunities for 
administrators (90%); and LDC staff were able to connect them to additional resources (87%). 


Classroom observation (A5—6). Almost half (45%) of administrators reported observing 
teachers implementing LDC three to five times during the year, while 35% observed one to two 
times, 17% observed six or more times, and 3% did not observe at all. These figures corroborate 
teachers’ reports of being infrequently observed, with 44% of teachers reporting that they were 
not observed at all during the school year, and 32% of teachers reporting one observation. In 
general, classroom observation was not a frequently used support structure. 


Almost all administrators (93%) believed that LDC modules were moderately or very 
effective in developing students’ literacy skills. 


Impact on teacher practice (A7). Administrators observed significant improvement in 
certain areas of teacher practice: 100% believed teachers had improved moderately or a great 
deal in selecting focus standards; 100% in identifying necessary skills to complete the writing 
assignment; 93% in creating standards-driven writing tasks; and 82% in assessing students’ 
progress as they completed the writing task. 


The majority of administrators believed teachers had improved on the following practices: 
75% in creating daily lessons to teach writing skills; 72% in using evidence of student 
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performance to shape instructional decisions; 68% in tracking and analyzing student progress in 
a systematic way; and 64% in identifying patterns of student understandings or misconceptions. 


These findings were similar to teachers’ self-reports of their own improvement in practice 
in that the two areas of most difficulty involved analyzing evidence of student progress. Areas 
of improvement most commonly reported by both groups involved selecting appropriate 
standards and skills. 


Impact on student learning (A8). Administrators were almost universally positive about 
the effects of LDC on students. Almost all (97%) agreed or strongly agreed that LDC helped 
students develop reading skills; LDC modules effectively supported students’ writing (97%); LDC 
supported students’ college and career-ready skills (96%); LDC improved students’ content 
knowledge (93%); LDC improved students’ writing quality (93%); LDC helped students 
understand components of the writing task (93%); LDC helped students’ overall literacy 
performance (93%); and LDC helped students develop speaking and listening skills (86%). 


Administrator leadership role (A9). Almost all administrators felt they played an active 
role in LDC implementation: 96% agreed or strongly agreed that they made changes to school 
schedules to accommodate LDC PLC time; 92% met regularly with the LDC project liaison; 89% 
were able to shape LDC implementation at their schools; and 71% led discussions about how to 
expand LDC implementation in future years. 


Alignment (A10). Administrators reflected on how well LDC aligned with other school 
initiatives, programs, and curricula (see Figure 3.7). Almost all (97%) agreed or strongly agreed 
that their school connected LDC to specific schoolwide goals; LDC helped teachers create 
writing assignments to use within current curricula (96%); LDC helped students prepare for 
state assessments (96%); LDC complemented other initiatives at the school (93%); and LDC was 
a strategy for implementing state college and career-ready standards (89%). Less than half 
(43%) believed that the time spent on LDC interfered with other initiatives, and 50% reported 
that it was difficult for teachers to focus on LDC because of other competing priorities. 
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It was difficult for teachers to focus on LDC 
because of other competing priorities at the 
school. (n=28) 


LDC helped prepare students in my school for 
current state assessments. (n=28) 


The time spent implementing LDC interfered 
with other important initiatives at my 
school. (n=28) 

LDC was a strategy for implementing my 
state’s College- and Career-Ready Standards. 
(n=28) 


LDC complemented other initiatives taking 
place in my school. (n=28) 


LDC helped teachers create writing 
assignments to use within their current 
curricula. (n=28) 


Our school connected LDC implementation to 
our specific schoolwide goals. (n=28) 
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Figure 3.7. LDC alignment (A10). 


Scale-up and sustainability (A11). The outlook for LDC implementation in future years 
was generally positive, according to administrators as reported in Figure 3.8. Almost all (93%) 
agreed or strongly agreed that teachers and administrators were committed to sustaining LDC; 
89% expected most teachers to continue the following year; and 75% expected the LDC PLC to 
increase in size. Notably, 79% of administrators observed new collaborations across grades 
and/or subjects, while only 38% of project liaisons reported the same. Almost half of 
administrators (39%) observed teachers who were not in LDC using the LDC planning process 
and tools, while only 20% of project liaisons observed this. 


eS) 


9 


100% 9 
6 390% 93% 
90% 
79% 


80% 75% 
70% 
60% 54% 
50% 
39% 
40% 
30% 
ie) 
20% 14% 18% 
10% 4% ie ie i 4% 4% eg 
ee a = ] o_o | 
| expect that Teachers at my As a result of Teachers and | expect our LDC 
most teachers school who were LDC, new administrators at PLC to increase 
participating in not part of the collaborations my school are’ in size next year. 
LDC this year willLDC PLC meetings across grades committed to (n=28) 


continue to do so- used the LDC and/or subjects sustaining the 
next year. (n=28) planning process were created or LDC initiative. 


and/or LDC are being (n=28) 
CoreTools. launched at my 
(n=28) school. (n=28) 


BAgree/ Strongly Agree O Disagree/Strongly disagree B Don’t know 
Figure 3.8. Scale-up and sustainability (A11). 


District support (A12). Administrators’ perceptions of district support for LDC were 
almost evenly divided between seeing support and not having knowledge of whether or not the 
district supported LDC. Over half of administrators (57%) agreed or strongly agreed that district 
leaders supported implementation of LDC; 46% said district leaders were interested in 
spreading LDC to additional schools; 43% reported district leaders had a firm understanding of 
LDC; 39% felt district professional development efforts were aligned with LDC; and 39% 
observed that district leaders visited the school to discuss LDC. 


3.4 Open-Ended Responses for All Participants 


Each survey respondent had the opportunity to provide open-ended feedback in response 
to three questions: (1) What supports did you find the most useful and why? (2) What supports 
were not helpful and why? and (3) In what ways could LDC implementation be improved in your 
school in the future? 


Below are the summary findings across all three groups of respondents, including 162 
teachers (including 25 project liaisons), 27 administrators (including two project liaisons), and 
two project liaisons who were neither teachers nor principals. Thirty-nine teachers (24%) 
reported they found all supports helpful, as did 15 administrators (56%). It is important to note 
that the majority of responses to all survey questions was positive; the specific concerns 
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described below belong to a minority of respondents who felt strongly enough to elaborate in 
the open-ended response section. 


Professional learning community. Weekly meetings with the LDC PLC received 12 positive 
and two negative mentions. The two negative comments were that combining subject areas 
and grades was not an efficient use of time. Positive comments were all about appreciating the 
opportunity to collaborate with other teachers. 


The most common suggestion for improving LDC implementation in the future was to 
provide more protected time for the PLC. Time was mentioned at least 54 times, which 
included requests for more time and support to both plan and implement modules (not 
necessarily within the PLC time) as well as suggestions about pacing and implementation. In 
particular, 11 teachers specifically requested support in watching others implement modules or 
being provided with “co-teachable opportunities” to provide peer feedback about 
implementation. 


About a dozen comments were made about needing an earlier start or more support to 
get the implementation off the ground. A handful of teachers specifically requested summer 
training and earlier start dates. 


Math and science teachers would prefer to have their own PLC, as they felt much of the 
ELA content was not relevant to them. In general, there was a sense that more specialized 
groups within PLCs would be more helpful. Several teachers commented that combining subject 
areas and grade levels contributed to boredom and disengagement because a good portion of 
the content was not relevant to them. 


Another common request mentioned by 25 respondents was to add more teachers to the 
initiative: across disciplines, grade levels, the entire school, and new teachers. 


The use of technology during and outside of the PLC was mentioned negatively 30 times, 
almost always about the difficulty of using the Zoom software. When the videoconferencing 
software and hardware did not function properly (whether it was on the school side or the 
coach side) participants felt it was a waste of time, awkward, and “painful.” There were further 
difficulties with technology specific to CoreTools, which are detailed in that section. 


Coach (in-person). In-person coaching encompassed several support types, including 
virtual coaches’ site visits and assistance from the district liaison and LDC liaison. A few teachers 
also included support from project liaisons in this category. One administrator mentioned 
paying for additional coach visits and felt it was well worth the cost. There were 45 comments 
indicating that in-person coaching was one of the most useful supports, and 31 comments 
indicating that it was insufficient. These negative comments were universally that the PLC did 
not receive enough or any in-person coaching, not about the quality of the in-person coaching. 
On the positive side, several respondents said there was no substitute for in-person support. 
For at least a handful of respondents, in-person coaching was a necessary component of 
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successful LDC implementation. Those that were more specific asked for help with seeing 
model lessons implemented in real time and in-person, individualized support. 


Many respondents applied their positive comments to both virtual and in-person coaching 
supports. One teacher wrote 


| found the virtual coaching and in-person coaching the most useful 
because my coach was able to help me reshape or retool my modules to 
fit my classes. Whenever | had a question or concern, my coach helped 
me find a clear path to helping my students meet high expectations on 
difficult writing assignments. 


Whether in person or virtually, teachers felt strongly about receiving feedback, which was 
mentioned 25 times. Teachers appreciated feedback in any format, whether via individual 
Zoom conferences, during PLC conferences, via email, or in the CoreTools and Teaching Channel 
platforms. 


Coach (virtual). LDC coaches were the most frequently mentioned item, with 70 
comments that they were one of the most useful supports and 31 comments that they were 
not helpful. These included comments about Zoom videoconference meetings with the entire 
PLC, individual Zoom meetings with coaches, email contact, and written feedback via CoreTools 
and Teaching Channel. Negative comments about virtual coaching that focused purely on 
technical issues were included in the PLC technology section, and not duplicated in this 
category. 


Among the positive mentions, respondents praised coaches’ feedback and guidance in 
navigating LDC website materials like CoreTools, the library of modules, and the online courses. 
They appreciated the ability to ask questions and receive immediate feedback. They also 
appreciated one-on-one videoconferences with coaches and written feedback via CoreTools, 
Teaching Channel, and via email. 


Negative comments included that the coaches were unengaging, impersonal, uninformed 
about local standards, or mismatched by grade or subject matter. The meetings went too fast, 
and several respondents just felt uncomfortable with the medium and preferred in-person, 
more traditional methods. Notably, among suggestions for improvement, 14 comments were 
made about requesting more in-person coach visits, while only one participant requested more 
video chats. 


“4 


CoreTools. The most common reaction to CoreTools was exemplified by this comment: 
really enjoyed using CoreTools once | learned how to use it.” CoreTools had 71 positive 
comments and 52 negative comments. 


Even among the positive comments, there was an acknowledgment that the learning 
curve was steep and barriers had to be overcome. The three most common barriers to 
mastering use of CoreTools were time to explore, guidance on features, and ability to navigate 
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easily. A handful of teachers did not find it useful for their specialties due to lack of content for 
their subject or student group: lower grades, mathematics, Spanish, English language learners, 
and students with disabilities. Specific features of CoreTools that were reviewed positively were 
the ability to receive coach feedback, help with aligning standards to tasks, and the “well 
planned information architecture.” 


Among suggestions for improvement, nine teachers felt the LDC module creation process 
was too fragmented and the interface too difficult. They requested more support, especially at 
the beginning. On the other hand, a small group of teachers protested that the templates were 
too simplistic (“basic checklists,” “cookie-cutter,” “filling in the blanks”) and would prefer “more 
complex tasks.” 


Another distinction between participants that came through in suggestions for 
improvement to CoreTools was the disparity in comfort with technology. It was frustrating for 
teachers who had higher facility with the website to sit through basic navigation tutorials. It 
was frustrating for teachers with lower technological knowledge to be rushed through the 
material. 


A small but vocal number of teachers were unhappy with the lack of materials on the 
website that related directly to their contexts; these included teachers who taught math, 
science, English language learners, younger students, and students with disabilities. 


Online courses. Online courses received seven positive comments and 22 negative 
comments. The single positive commenter who elaborated said that she appreciated the ability 
to go at her own pace. Among the many negative elaborations, the most common issues were 
that course content was difficult to navigate, too time consuming, redundant, “too broad to be 
helpful,” and difficult to read (“walls of text”). The embedded videos did not work for everyone 
due to school security measures. 


Teaching Channel. Teaching Channel resources, including videos and discussions, were 
mentioned four times as helpful and seven times as unhelpful. A few teachers provided their 
reasons: engaging teachers was difficult because of competing initiatives; there was confusion 
about using two different websites (Teaching Channel and CoreTools); and their school 
curricula were already aligned to Common Core and so they had no need for Teaching Channel 
discussions. 


3.5 Summary of Results 


This survey captured the responses of 81% of all participants in the district’s first year of 
implementation. In addition to answering closed-ended questions, most of the respondents 
also provided narrative comments about aspects of the program they felt were useful as well as 
those that could use improvement. Generally, respondents provided positive feedback. Overall, 
the survey results suggest the following: 
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Across all participant groups, survey responses showed positive attitudes toward LDC. 
All measures of satisfaction or improvement were rated positively by more than half 
of respondents. 


Seventy percent of teachers and 93% of administrators agreed that LDC helped 
improve students’ literacy performance. 


LDC coaches received almost unanimous positive feedback, with 95% of teachers 
reporting that their coaches gave them appropriate and timely feedback and support. 
But there are a few negative comments about coaches in the open-ended responses. 


Project liaisons were almost universally reported as very approachable, effective, and 
knowledgeable. Almost all teachers reported that their administrators encouraged 
LDC participation at the school, though only 58% of teachers reported receiving 
feedback from administrators about LDC and 44% reported never being observed 
while teaching an LDC task. 


Almost three quarters of teachers felt they had sufficient time to meet during 
professional learning community (PLC) sessions and 86% reported that administrators 
protected that time. However, the most frequently requested modification for future 
years was for more protected time during and outside of PLCs to plan modules, 
implement, and receive feedback about implementation. 


Almost two thirds of teachers expressed interest in learning more about how to lead 
LDC implementation at their schools. Over 80% of project liaisons and administrators 
expected their teachers to continue with LDC the following year. Both groups 
observed that teachers at the school who did not participate in the LDC PLC generally 
did not adopt LDC methods or materials on their own. 


While 79% of teachers agreed that their school had adequate technology to support 
LDC implementation, a common complaint by participants was the difficulty in 
successfully conducting Zoom meetings. 


With regard to creating modules, teachers were most likely to report success with 
selecting focus standards, creating the writing assignment, identifying skills, and 
providing multiple opportunities to engage students with the materials. They were 
less likely to report success with selecting texts, making connections to previous or 
future work, creating daily lessons, and planning a variety of assignments. 


With regard to implementing LDC in the classroom, teachers were most likely to 
report success with engaging students in productive struggle, differentiating 
instruction, and locating evidence of progress in student work. They were slightly less 
likely to report success in using evidence to modify plans, providing feedback, and 
systematically collecting information about student learning. 


When asked about LDC impact on students, teachers reported positive impact on 
writing quality, college and career readiness skills, overall literacy performance, 
reading skills, content knowledge, and speaking and listening skills. 
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This year’s survey responses were constantly positive across all dimensions and 
participant groups. Suggestions and recommendations provided in this report were primarily 
about adapting to and ameliorating district constraints, such as protected time for meetings 
and technological limitations. 
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4.0 Analyses of LDC CoreTools Data 


In the following section, we report on how participants interacted with LDC’s CoreTools 
online system. We begin in Section 4.1 by presenting participation rates for key CoreTools 
activities, including creating a user account, viewing modules and mini-tasks, editing modules 
and mini-tasks, and commenting on modules and mini-tasks. In Section 4.2, we dig deeper into 
CoreTools viewing, editing, and commenting by sharing descriptive statistics for all i3 CoreTools 
users followed by results broken down by key subgroups (including LDC role, school level, and 
content area taught). In Section 4.3, we compare the level of engagement for i3 CoreTools 
users who completed and taught a full-length module to those users who did not complete and 


teach a module during the school year. 


4.1 CoreTools Activity Participation Rates 


The four key CoreTools activities we examined are creating a user account, viewing 
modules and mini-tasks, editing modules and mini-tasks, and commenting on modules and 
mini-tasks. Among the 265 CoreTools users, 218 were teachers, and 47 of them were 
administrators, with some individuals in each category acting as project liaisons. As seen in 
Table 4.1, nearly all participants used CoreTools to at least some degree. Eighty-nine percent of 
all participants created a user account, 83% of participants viewed modules or mini-tasks, 77% 
edited modules or mini-tasks, and 45% of them commented on modules or mini-tasks. 


Table 4.1 


CoreTools Feature Participation Rates: 2016-2017 


Number and Numberand Number and% 
Number % of % of of participants 
and % of participants —_ participants who 
participants whoviewed whoediteda commented on 
Number of with user a module or module or a module or 
Participant type participants accounts mini-task mini-task mini-task 
Teacher 188 175 (93%) 171 (91%) 166 (88%) 104 (55%) 
Teacher/project liaison 30 27 (90%) 27 (90%) 27 (90%) 14 (47%) 
Total teachers 218 202 (93%) 198 (91%) 193 (89%) 118 (54%) 
Project liaison? 1 1 (100%) 1 (100%) 0 (0%) 0 (0%) 
Administrator 42 28 (67%) 19 (45%) 8 (19%) 1 (2%) 
Administrator/project 4 4 (100%) 3 (75%) 3 (75%) 0 (0%) 
liaison 
Total administrators ° 47 33 (70%) 23 (49%) 11 (23%) 1 (2%) 
Total participants 265 235 (89%) 221 (83%) 204 (77%) 119 (45%) 


*We categorize the coaches and coordinators who completed the project liaison survey as administrators. 
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In addition to displaying participation rates on key CoreTools activities for all i3 
participants, Table 4.1 also displays subgroup results for participants playing different roles in 
the i3 implementation. These subgroups mirror the five groups to whom we administered 
surveys: teachers, project liaisons, administrators, teacher/project liaisons, and 
administrator/project liaisons. Teachers had higher participation rates than the administrators, 
as expected. 


Nearly 90% of participants created a CoreTools user account. Significantly, even 
administrators who did not play a project liaison role created user accounts in high numbers 
(two out of three did so). This suggests that the vast majority of administrators overseeing LDC 
PLCs had at least some familiarity with the online platform. 


Likewise, large majorities of PLC teachers and project liaisons (between 75% and 100% 
depending on the participant role) viewed modules and/or mini-tasks in CoreTools. 
Administrators who did not play a project liaison role were considerably less likely to engage in 
this way, although almost half still viewed at least one module or mini-task. 


We consider editing modules and mini-tasks to be the key indicator of deep engagement 
with the CoreTools module building platform. Eighty-nine percent of participating teachers 
edited at least one module or mini-task. Administrators were much less likely than teachers to 
edit materials in CoreTools with less than a quarter having done so. 


Overall, adding comments to modules or mini-tasks was a much less common activity, 
with about half of teachers having commented and only one administrator having done so. This 
perhaps should not be surprising, as the i3 model set an expectation that coaches provide 
feedback to teachers via comments, but did not require teachers to comment back or reflect on 
peers’ work. 


4.2 Engagement With Key CoreTools Activities 


In this section, we describe participants’ level of engagement with three key CoreTools 
activities: viewing modules and mini-tasks, editing modules and mini-tasks, and commenting on 
modules and mini-tasks. Descriptive statistics are reported for all participants, as well as a 
number of subgroups. Those subgroups capture the role the individual played in LDC (teacher, 
project liaison, and administrator), the level of the school at which the participant worked 
(elementary, K-8, middle, 6-12, or high), and in the case of teachers, the content area taught 
(elementary/multiple subjects, secondary ELA, secondary history/social studies, secondary 
science, secondary math, special education, and English as a second language). As noted earlier, 
some participants played multiple roles in the intervention, so the teacher, project liaison, and 
administrator groups overlap to some degree as they do in our survey analysis. Descriptive 
statistics are only reported for groups with five or more members. Finally, note that for each of 
the three analyses below, participants who did not engage in the activity are excluded from the 
analysis (i.e., the analyses do not include any observations with zero views, edits, or 
comments). As such, in each case, the results describe the behavior of participants who 
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engaged in the particular activity at least once. The samples for the viewing, editing, and 
commenting analyses are 221, 204, and 119 respectively (see bottom row of Table 4.1). 


Module/mini-task viewing. In Table 4.2, we present descriptive statistics on how many 
times i3 participants viewed modules and mini-tasks, both overall and by subgroup. We present 
the minimum number of views, the maximum number of views, the mean number of views, the 
standard deviation, and the sum of total views across all participants. Overall the average 
participant viewed modules or mini-tasks over 54 times, while the range was from one view to 
481 views. The standard deviation of 61.9 suggests that about two thirds of all participants 
viewed modules or mini-tasks between 0 and 115 times (with the remaining third viewing a 
greater number of times). 
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Table 4.2 


Descriptive Statistics for the Number of Times a Participant Viewed a Module or Mini-Task in CoreTools, 
by Participant Subgroup 


Subgroup n Min Max M SD Sum 
All participants 221 1 481 54.5 61.9 12046 
Participant role 
All teachers 198 1 481 59.3 63.5 11744 
All project liaisons 31 2 246 63.9 55.7 1982 
All administrators 23 1 57 13.1 15.8 302 
School level 
Elementary school level participants 12 10 341 110.9 83.7 1331 
K-8 school level participants 24 1 481 70.2 99.3 1684 
Middle school level participants 119 1 196 43.2 36.3 5141 
6-12 school level participants 27 3 353 87.9 95.3 2373 
High school level participants 39 2 197 38.9 35.8 1517 


Content area taught 


Taught elementary/multiple subjects 23 11 341 85.3 72.7 1962 
Taught secondary ELA 72 6 481 72.8 80.5 5244 
Taught secondary social 33 8 481 73.4 84.9 2423 
studies/history 

Taught secondary science 37 2 129 41.1 32.2 1519 
Taught secondary math 14 14 129 48.4 31.9 677 
Taught special education 23 1 222 68.9 54.6 1584 
Taught ESL 8 2 66 35.6 25.4 285 


As would be expected, project liaisons on average had the greatest number of views, 
while administrators viewed the least number of times. Teachers viewed only slightly fewer 
modules and mini-tasks than project liaisons. There was considerable variation in viewing 
behavior across participant subgroups within these categories. Participants in elementary, K-8, 
and 6-12 schools viewed considerably more modules and mini-tasks on average than their 
middle and high school peers. Elementary school teachers on average viewed a greater number 
of modules and mini-tasks than their secondary teacher peers. At the secondary level, ELA and 
social studies/history teachers viewed more modules and mini-tasks than math and science 
teachers. ESL teachers showed the least engagement of the content area subgroups. 
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Module/mini-task editing. On average, participants who engaged in editing modules or 
mini-tasks did so 32 times over the course of the school year. There was a wide range of 
engagement from editing once to making 241 edits to modules and mini-tasks. Two thirds of 
participants edited between zero and 64 times. Table 4.3 reports the descriptive results. 


Table 4.3 


Descriptive Statistics for the Number of Times a Participant Edited a Module or Mini-Task in CoreTools, 
by Participant Subgroup 


Subgroup n Min Max M SD Sum 
All participants 204 1 241 31.6 32.2 6438 
Participant role 
All teachers 193 1 241 32.9 32.6 6353 
All project liaisons 30 5 123 40.5 33.6 1214 
All administrators 11 1 20 7.7 5.9 85 
School level 
Elementary school level participants 11 10 118 51.6 39.3 568 
K-8 school level participants 21 1 241 43.2 52.1 908 
Middle school level participants 109 1 123 22.9 21.5 2499 
6-12 school level participants 25 1 158 52.9 39.6 1322 
High school level participants 38 3 135 30.0 26.8 1141 


Content area taught 


Taught elementary/multiple subjects 23 6 118 40.1 33.3 922 
Taught secondary ELA 71 1 241 38.6 41.5 2739 
Taught secondary social 33 1 241 39.1 46.2 1291 
studies/history 

Taught secondary science 35 1 82 21.2 18.0 741 
Taught aecondary math 14 5 48 20.5 14.3 287 
Taught special education 22 7 122 44.7 30.5 984 
Taught ESL 7 3 33 19.4 11.7 136 


As with page viewing results, project liaisons edited more than teachers, and 
administrators edited much less than teachers. Differences between participant school-level 
groups in the number of edits also followed similar patterns to module/mini-task viewing, with 
elementary, K-8, and 6—12 participants making a greater number of edits than their middle and 
high school level peers. Secondary ELA and social studies/history teachers edited more than 
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secondary math and science teachers. Again, ESL teachers exhibited the lowest level of 
engagement. Despite mean differences, there was great variation across participants within 
each of the subgroups. 


Module/mini-task commenting. Only 119 participants, representing less than half of all 
participants, commented on modules or mini-tasks in the past school year. Within that highly 
engaged group, participants commented between one and 43 times, and on average five times. 
All but one of the commenters were teachers, and a little over 10% of the group were project 
liaisons as well. Those participants who played a project liaison role commented more on 
average than teachers not playing that role. Differences between subgroups followed similar 
patterns to viewing and editing, with elementary level teachers making the highest number of 
comments among school-level groups, and secondary ELA and social studies/history teachers 
commenting more than secondary math and science teachers. 


Table 4.4 


Descriptive Statistics for the Number of Times a Participant Commented on a Module or Mini-Task in 
CoreTools, by Participant Subgroup 


Subgroup n Min Max M SD Sum 
All participants 119 1 43 5.3 6.5 625 
Participant Role 
All teachers 118 1 43 5.3 6.5 620 
All project liaisons 14 1 43 8.5 11.3 119 
School! Level 
Elementary school level participants 11 1 43 10.8 14.8 119 
K-8 school level participants 13 1 26 5.3 7.2 69 
Middle school level participants 46 1 19 4.0 4.3 183 
6-12 school level participants 24 1 18 6.9 5.2 166 
High school level participants 25 1 8 3.5 2.3 88 


Content area taught 


Taught elementary/multiple subjects 17 1 43 7.7 12.5 131 
Taught secondary ELA 47 1 26 5.6 5.4 265 
Taught secondary social 19 1 26 6.1 7.4 115 
studies/history 

Taught secondary science 17 1 11 3.4 3.0 58 
Taught secondary math 6 1 6 2.8 2.2 17 
Taught special education 19 1 18 5.1 49 97 
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4.3 CoreTools Engagement as an Implementation Variable 


To evaluate the validity of CoreTools engagement as an indicator of LDC implementation, 
we examined the relationship between the three CoreTools engagement measures and module 
implementation. As described in the next chapter, CRESST identified full-length modules that i3 
participants created and/or adapted and taught in their classrooms. We took the presence of 
uploaded student work as evidence of teachers having taught the module in their classrooms. 
This parameter yielded a sample of 169 modules, associated with 122 teachers and three 
administrators. Figure 4.1 displays the mean number of CoreTools views, edits, and comments 
for the 122 teachers who completed and taught full-length LDC modules and their 96 peers for 
whom we do not have evidence of full module implementation. As shown, teachers who 
completed and taught full-length LDC modules exhibited over twice as much viewing and 
editing activity as their peers, and over five times the commenting activity. The results suggest 
that participants who engage deeply with the module building platform are more likely to 
complete and teach LDC modules. 
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Teachers who did not complete and teach modules (n=96) 


Figure 4.1. Mean number of CoreTools views, edits, and comments made by 
teachers who did and did not complete and teach modules. 
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5.0 Module Artifact Analysis 


Table 5.1 presents mean scores and standard deviations for all modules analyzed. Among 
the 114 modules scored, 20 were at the elementary level, and 94 were at the secondary level. 
As previously stated, each module was rated using a 5-point scale to measure (a) the 
effectiveness of the writing task or objective(s) for the module, (b) the module’s alignment to 
the standards, (c) fidelity to the four stages of LDC instructional practice (i.e., preparation for 
the task, reading process, transition to writing, and writing process), (d) the quality of the 
instructional strategies, (e) coherence and clarity of the module, and (f) the overall quality of 
the module (see Appendix A for full descriptions of the dimensions). 


Table 5.1 
Means by School Level 
Elementary Secondary All modules 

Dimension n M (SD) n M (SD) n M (SD) 
1. Effective writing task 20 3.71.4) 94 3.5 (1.1) 114 3.5 (1.1) 
2. Standards alignment 20 2.8 (1.6) 94 3.3 (1.2) 114 3.2 (1.3) 
3. Fidelity to LDC module instruction 20 2.4 (1.8) 94 2.8 (1.5) 114 2.7 (1.6) 
4. Quality instructional strategies 20 2.4 (2.0) 94 2.5 (1.5) 114 2.4 (1.6) 
5. Coherence and clarity of module 20 2.1 (2.0) 94 2.5 (1.5) 114 2.5 (1.6) 
6. Overall impression 20 2.4 (1.6) 94 2.6 (1.4) 114 2.6 (1.5) 


As can be seen, modules at the secondary level received somewhat higher mean ratings 
than did the modules at the elementary level on nearly all of the dimensions. More specifically, 
mean ratings for the secondary modules were in the 3s for Dimensions 1 and 2, which focused 
on the effective writing task and standards alignment, but were in the mid to high 2s for the 
remaining dimensions. In contrast, the only dimension in which elementary modules had a 
mean greater than 3 was for the effective writing task. All of the remaining dimensions ranged 
from the low to high 2s. In other words, teachers were moderately successful in regards to 
Dimension 1, and Dimension 2 when teaching at the secondary level. Otherwise, dimensions 
were on average only barely present. 


For the remainder of this report, we present results for the elementary and secondary 
modules separately. This will include further descriptive results, results from generalizability 
theory analyses, as well as factor analyses for the elementary modules. Because of the small 
sample sizes, reported findings are limited to descriptive results for the elementary modules as 
well as the secondary science and social studies modules; generalizability and factor analyses 
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for these subsamples will be conducted at a later time once a larger sample of eligible modules 
has been collected. 


5.1 Elementary Module Results 


Table 5.2 presents descriptive results for the elementary modules by content area. A few 
comparative observations can be made. First, when looking across content areas, modules 
tended to receive the highest ratings on Dimension 1, concerning the effective writing task. 
While the number of modules was very small, it also appears that science and social studies 
modules were more successful than ELA modules with regard to most dimensions; science 
modules received on average ratings of 3 or greater for all dimensions; and social studies 
modules received similar ratings on average for all dimensions (with the exception of coherence 
and clarity of the module). In contrast, ELA modules tended to receive mean ratings from the 
high 1s to the mid 2s, which indicates that these dimensions were not or were barely present or 
realized. Again, given the very small subsamples, these results should be interpreted with 
caution. 


Table 5.2 


Descriptive Statistics for the Elementary Modules by Content Area 


ELA Science Social studies 

Dimension n M (SD) n M (SD) n M (SD) 

1. Effective writing task 14 3.4 (1.6) 3 4.7 (0.6) 3 4.0 (1.0) 
2. Standards alignment 14 2.4 (1.6) 3 3.3 (1.2) 3 3.7 (1.5) 
3. Fidelity to LDC module instruction 14 2.0 (1.5) 3 3.3 (2.1) 3 3.0 (2.7) 
4. Quality instructional strategies 14 1.9 (1.8) 3 4.0 (1.7) 3 3.0 (2.7) 
5. Coherence and clarity of module 14 1.8 (1.8) 3 3.0 (2.7) 3 2.7 (2.5) 
6. Overall impression 14 2.1 (1.4) 3 3.3 (2.1) 3 3.0 (2.0) 


5.2 Secondary Module Results 


Table 5.3 presents descriptive results for the secondary modules by content area. While 
some of the subgroups are small (math in particular), some general observations can be made. 
First, modules tended to receive higher ratings for Dimensions 1 and 2, which focus on the 
effective writing task and standards alignment, with mean ratings of 3 to 4. This indicates that 
these two dimensions were often moderately to sufficiently present or realized. Second, ratings 
were generally low for Dimensions 3 through 6, with means primarily in the 2s, which indicates 
that these traits were on average barely present. Furthermore, the few math modules 
submitted received very low means which indicates that Dimensions 3 through 6 were not 
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present or realized. These results seem in line with comments made by raters during 
debriefings stating that the potential of the module, as seen in “Section 1: What Task,” was not 
always realized through the remainder of the writeup covering the instructional ladder. 
Furthermore, expert teachers who rated the few math modules indicated that the LDC module 
structure was not well aligned to the content area. 


Table 5.3 
Descriptive Statistics for the Secondary Modules by Content Area 


ELA Math Science Social studies 

Dimension n M (SD) n  M/(SD) n M (SD) n M (SD) 

1. Effective writing task 61 3.4(1.0) 4  3.5(1.0) 16 4.0 (1.1) 133.4 (1.1) 
2. Standards alignment 61 3.2 (1.1) 4 3.8 (1.5) 16 4.0 (1.3) 13 3.0 (1.3) 


3. Fidelity to LDC module 61 2.9(1.5) 4 1.0 (0.8) 16 3.1 (1.6) 13. 2.7 (1.7) 
instruction 


4. Quality instructional 61 2.6(1.5) 4 0.8 (1.0) 16 2.4 (1.5) 13.2.7 (1.7) 
strategies 


5. Coherence and clarityof 61 2.6(1.5) 4 0.5 (0.6) 16 =2.6 (1.6) 13.2.5 (1.8) 
module 


6. Overall impression 61 = 2.7 (1.4) 4 1.3 (1.0) 16 =2.8 (1.5) 13.2.5 (1.8) 


Generalizability theory. As previously noted, generalizability analyses were limited to the 
secondary ELA sample because of issues with small sample sizes. Tables 5.4 and 5.5 present 
results from the two-faceted and one-faceted models, which examine error across and within 
dimensions respectively. As would be the goal of any rating session, most of the variation found 
in the secondary ELA ratings seems to be due either directly to the modules created by the 
teachers (44%) or to differing quality across the modules by dimensions (40%). Furthermore, 
less than 3% of the variation could be attributed either directly to the raters or to an interaction 
between raters and dimensions or modules. What is of concern, though, is that 8% of the 
variation for the ELA ratings was unexplained. 
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Table 5.4 


Generalizability Study of the Secondary ELA Module 
Ratings Across Dimensions 


Source Var. % 
Module (o2t) 0.79 44.4 
Rater (o2r) 0.02 0.9 
Dimension (o2d) 0.08 4.6 
Module x Dimension (o2td) 0.71 39.8 
Rater x Dimension (o2rd) 0.03 1.9 
Module x Rater (o2tr) 0.00 0.0 
Error (o2trd,e) 0.15 8.5 


Note. Negative estimates of variance were changed to zero 
in order to calculate percentages (see Shavelson & Webb, 
1991). 


As previously mentioned, we also used a one-faceted design to further disentangle the 
variance that was due either directly or through an interaction with the dimensions (see Table 
5.5). As we would hope, the vast majority of the variation in most of the dimensions could be 
attributed directly to the modules created by the secondary teachers. In this case, more than 
90% of the variation for Dimensions 3 through 6 were due to differences in the modules 
created by the teachers. In contrast, about two thirds of the variation for Dimensions 1 and 2, 
which focus on the effective writing task and standards alignment, were attributable to the 
modules. For Dimension 1, the remaining variation was primarily due to differences amongst 
raters (12%) or differences in raters across modules (11%). For Dimension 2, almost all of the 
remaining variation was due to differences in raters across modules (32%). It is difficult to know 
whether this result is an artifact of having the raters for the secondary ELA panel only partially 
crossed—that is only eight of the 55 modules that were used for calibration purposes were 
rated by all three expert teachers—or was the result of issues with training and/or anchor 
points concerning standards alignment. 
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Table 5.5 
Generalizability Studies of the Secondary ELA Module Ratings for Each Dimension 


Module Module x Error 
(o2t) Rater (o2r) Rater (o2tr) (o2trd,e) 

Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 0.70 70.9 0.12 12.0 0.11 11.2 0.06 5.9 
2. Standards alignment 0.88 67.0 0.02 1.3 0.42 31.8 -0.16 0.0 
3. Fidelity to LDC module 1.96 94.0 0.03 1.4 0.02 1.0 0.08 3.7 
instruction 
4. Quality instructional strategies 1.95 90.4 0.04 2.0 0.14 6.4 0.03 1.2 
5. Coherence and clarity of 1.90 90.5 0.10 4.5 0.05 2.5 0.05 2.5 
module 
6. Overall impression 1.68 92.6 0.07 3.8 0.06 3.5 0.00 0.1 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see Shavelson & 
Webb, 1991). 


Factor analysis. Table 5.6 presents the principal component solutions extracted from the 
teacher scores for each dimension for the secondary sample. The model for the secondary ELA 
sample loaded on two factors, with the first factor accounting for more than half of the 
variance in ratings across modules (56.76%). The analysis suggests that there were two factors 
or underlying traits, with the first consisting of Dimensions 1 and 2 and the second consisting of 
Dimensions 3 through 6. 


Table 5.6 
Principal Component Analysis of the Secondary ELA Module Ratings (n = 61) 


Dimension Factor 1 Factor 2 
1. Effective writing task 0.42 0.71 
2. Standards alignment 0.43 0.72 
3. Fidelity to LDC module instruction 0.77 0.02 
4. Quality instructional strategies 0.88 -0.26 
5. Coherence and clarity of module 0.89 -0.22 
6. Overall impression 0.94 -0.21 
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5.3 Qualitative Results 


Expert raters were asked to participate in debriefings in an attempt to shed light on their 
ratings and suggest potential ways to improve ratings in the future. These debriefings were 
conducted at the end of each set of rating sessions (e.g., elementary, secondary ELA, etc.). The 
following presents key findings concerning Dimensions 1 through 5, as well as our expert raters’ 
overall impressions. 


Dimension 1: Effective writing task. While this was often the highest scoring category, 
few teachers took full advantage of the opportunity to explain their writing task. Many teachers 
simply filled in mandatory sections of the teaching task, without providing the optional 
questions allowed at the beginning. Furthermore, many provided limited information or none 
at all in the optional student background section. Our expert raters felt strongly that this 
additional information was necessary to know what was expected of students and to determine 
what the expectations were for higher order thinking. 


Dimension 2: Standards alignment. Teachers were inconsistent in their specification of 
standards for the modules. First, some modules included standards primarily in the designated 
area in the “What Task” section of the module creator, other modules included them with the 
mini-tasks, and yet others included information on standards in more than one location. This 
inconsistency caused raters to spend more time trying to disentangle issues of alignment. It also 
should be noted that raters felt that teachers often lacked focus in specifying their standards, 
with some providing too many and others providing too few. In some cases, this was 
hypothesized to be an artifact of teachers adapting rather than completing original modules. 


Dimension 3: Fidelity to LDC module instruction. While ratings concerning fidelity to LDC 
module instruction varied greatly in quality, especially among the secondary modules, raters 
generally felt that teachers were more successful in the early rather than the later parts of the 
instruction. Students were given less instruction than considered necessary for the transition to 
writing and writing process skills. Furthermore, teachers used default mini-tasks that showed 
little adaptation or that did not align with the teaching task. 


Dimension 4: Quality instructional strategies. Feedback on the quality of instructional 
strategies tended to mimic what was discussed concerning Dimension 3. Raters believed that 
artifacts showed little or no evidence of how teachers were scaffolding critical thinking. Many 
of the modules used the same mini-tasks, the writing task as specified often did not match the 
task template or the student work samples, and many modules felt like they were “cut and 
pasted” together. 


Dimension 5: Coherence and clarity of module. As one of our raters noted, coherence 
and clarity of the modules were “almost always either great or terrible.” Raters noted a 
disconnect between the instructional strategies and what students produced. Furthermore, 
they considered it difficult to rate this dimension in cases where teachers provided little 
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student work. In the end, though, modules that raters felt that they could take and implement 
directly in their own classrooms tended to receive the highest ratings for this dimension. 


Dimension 6: Overall impressions. Original modules were typically thought to be of much 
higher quality than those that were adapted. These modules were easier to understand, 
seemed to be more thought out, and had fewer mismatches across the task sections. Raters 
also felt that because they lacked understanding of what actually was changed or customized in 
the adapted modules, they had a limited ability to judge the quality of the work. Our expert 
teachers suggested that if more teachers had completed the student background and teacher 
reflection sections, and had included more student work samples, this would have greatly 
helped in the rating process. In addition, raters suggested that the addition of a section to 
provide details about the students, such as information about English language learners or prior 
achievement, would have been helpful in determining the appropriateness of the task template 
and mini-tasks. 


5.4 Summary of Results 


While the elementary sample and science and social studies subsamples for the secondary 
module ratings were fairly small, and should therefore be interpreted with caution, some 
trends did emerge across modules. First, teachers were generally more successful regarding 
Dimensions 1 and 2 than they were with the other Dimensions. With the exception of 
Dimension 2 for the elementary ELA modules, all content area subgroups received ratings in the 
3s for these two dimensions, which indicates that the effective writing task and standards 
alignment were moderately present or realized. Second, results of the generalizability study 
provide evidence that secondary ELA raters did a good job, with most variation due directly to 
differences in quality across the modules (44.38%) or to differing quality across the modules by 
dimension (39.82%). Likewise, generalizability study results showed that almost all variation in 
ratings for Dimensions 3 through 6 were due to differences in the secondary ELA modules 
themselves (90.42% to 93.95%). In contrast, while the majority of variation for Dimensions 1 
and 2 were also due to differences in the secondary ELA modules (70.92% and 66.95%, 
respectively), moderate amounts of variation were also found by rater for Dimension 1 
(12.01%) and for an interaction between module and rater for Dimensions 1 and 2 (11.15% and 
31.80%). This echoed results from the factor analysis, with Dimensions 1 and 2 loading on one 
factor and Dimensions 3 through 6 loading on another factor. In total, both factors accounted 
for 56.76% of variation across the secondary ELA modules. 
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6.0 Student Outcome Analysis 


This section presents the student outcome analysis we conducted to evaluate LDC’s 
impact on student learning in the 2016-2017 school year. We begin by describing the process 
we used to define the LDC student sample and to construct a matching comparison sample. We 
then present descriptive statistics for the treatment and comparison groups. Finally, we report 
the estimated impact of LDC on students as measured by the New York State assessment scores 
in English language arts (ELA). 


6.1 LDC Sample and the Matching Process 


As described earlier, our LDC teacher sample included both elementary and middle school 
teachers in the study district. Separate sampling and analyses were conducted for these two 
groups of teachers, as described in Section 2.5 of this report (Analytical Approaches). 


Elementary sample. The eligible LDC student sample for elementary includes all students 
who were enrolled for a minimum of two thirds of the overall school days preceding the 2016- 
2017 school year test administration under the instruction of at least one of the participating 
LDC teachers, and who had available data both for prior and current year achievement scores 
and demographic characteristics. 


The comparison sample was selected via a multistage process. In the first stage, we 
identified comparison schools for each of the participating LDC schools, based on similarity in 
both prior achievement and demographics. Then within these comparison schools, we selected 
comparison students who were similar to each of the LDC students. We used the following 
variables to identify similar comparison schools: the percentage of students eligible for free or 
reduced price lunch, the percentage of African American students, mean prior student 
achievement in ELA, the attendance rate of teachers, the teachers’ active years of teaching 
experience, and the school grade span. At the student level, we used CEM and matched 
students exactly on grade level as well as whether they were categorized as Asian. We also 
matched closely on variables for Black, Hispanic, White, other ethnicity, free or reduced price 
lunch status, special education status, and on prior individual student achievement and the 
mean prior achievement of students’ core content class peers. Finally, students also were 
matched on the average years of teaching experience of their core content teachers. 


The 2016-2017 elementary sample prior to the CEM process included five schools, the 15 
elementary teachers who consented to participate in the evaluation study, and their 273 
students. After the student-level matching, our final elementary LDC sample included 234 
elementary students and retained nearly the same number of teachers and schools (see Table 
6.1). 


Prior to matching, the potential elementary comparison sample consisted of 781 schools, 
10,106 teachers, and 123,307 students. This comparison sample was substantially reduced 
during the first stage of matching, which identified the five schools that most closely resembled 
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each of the five LDC elementary schools. After student-level CEM, a workable analytic 
comparison sample consisted of 19 schools, 100 teachers, and 234 students (see Table 6.1). 


Table 6.1 
Before and After Matching Sample Sizes: Elementary School Analysis 


LDC sample Comparison sample 
School Teacher Student School Teacher Student 
Stage 1 
Before matching 5 15 273 781 10,106 123,307 
After matching 5 15 273 25 474 4,979 
Stage 2 
After matching 5 14 234 19 100 234 


Middle school sample. The eligible LDC student sample for middle school was more 
challenging to identify than the elementary school sample because students were taught by 
multiple teachers of different subjects, each of whom did or did not participate in LDC. Our first 
step in sampling was to define at each school the potential sample of LDC students. These were 
all students at the school who enrolled in core content courses for at least two thirds of the 
overall school days preceding the assessment period. We began by classifying core content 
classes in the three core content areas (ELA, social studies/history, and science) based on the 
statewide course catalog (SCED). In addition, to be part of the sample, prior and current 
achievement scores needed to be available for each student as they were used for selecting the 
middle school sampling. 


The resulting LDC sample included 20 schools, 104 teachers, and 3,562 students prior to 
the CEM process. After the CEM student-level matching, our final secondary LDC sample was 
reduced to 3,214 students, with all schools and teachers being retained (see Table 6.2). 
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Table 6.2 
Before and After Matching Sample Sizes: Middle School Analysis 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 20 104 3,562 519 10,131 161,047 
After matching 20 104 3,562 100 2,304 29,375 
Stage 2 
After matching 20 104 3,214 85 1,423 3,214 


Prior to matching, the potential comparison sample consisted of 519 schools, 10,131 
teachers and 161,047 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified the five schools that most closely resembled each of 
the 20 LDC schools. After student-level CEM, an analytic comparison sample of 85 schools, 
1,423 teachers, and 3,214 students was constructed (see Table 6.2). 


6.2 Descriptive Results on the Matched Analytic Samples 


Tables 6.3 and 6.4 present the student characteristics for the LDC student and comparison 
students, based on the final analytical samples for the elementary and secondary analyses 
respectively. Treatment and comparison samples matched very closely on all variables. Exact 
matching was achieved on some variables, and all demographic variables were within 4 
percentage points. For the prior achievement matching variable, we standardized students’ 
2016 New York State assessment scale scores at each grade level relative to district 
performance, based on the district mean and standard deviation for the ELA test at each grade 
level. Standardizing scores in this way enables us to easily compare our samples’ performance 
relative to the district and to compare scores across grades and years more easily. A 
standardized scale score of zero, for example, indicates that the student scored at the mean for 
all other students in the district who took the same test. A standardized scale score of 1.0 
meant that the student scored one standard deviation higher than the district mean. 
Conversely, a standardized scale score of -1.0 indicated that the student scored one standard 
deviation lower than the district mean. 


The final LDC elementary student sample after matching was composed largely of Black, 
Asian, and Hispanic students, with more than one third being of Black ethnicity, almost one 
third Asian, and about slightly over one quarter Hispanic (see Table 6.3). The sample was 
composed of fourth and fifth grade students and mostly of students with low socioeconomic 
status (88.5%). English language learners, special education students, and gifted students each 
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represented between 10 and 15% of the sample. Mean performance on the prior year 
academic assessment was very close to the districtwide performance levels in math and ELA. 


As shown in Table 6.4 the final LDC middle school student sample included a large 
proportion of Black and Hispanic students (87.2%), and a similarly large proportion of students 
with low socioeconomic backgrounds (86.1%). The sample was composed of sixth (16.1%), 
seventh (41.1%), and eighth grade students (42.8%). Special education students and gifted 
students each represented slightly less than one quarter of this sample, while there was about 
one half that many English language learners (12.8%). In addition, mean performance on the 
prior year academic assessment was somewhat lower for LDC students as compared to 
districtwide performance levels in math and ELA. 
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Table 6.3 


Student Characteristics of Elementary School Treatment and Comparison Groups After Coarsened Exact 
Matching, 2016-2017 School Year 


Comparison sample: 


LDC sample: Elementary Elementary 
Student characteristic (n = 234) (n = 234) 

Race/ethnicity 

Hispanic (%) 25.6 27.8 

Black (%) 38.0 34.6 

Asian (%) 30.8 30.8 

White (%) 3.0 4.3 

Other (%) 2.6 2.5 
Female (%) 47.9 48.7 
Special programs status 

Poverty (%) 88.5 90.4 

English language learner (%) 12.4 8.5 

Special education (%) 14.1 12.8 

Gifted (%) 14.5 13.2 
Student prior achievement 

Mean prior year math Z score -0.01 -0.03 

Mean prior year ELA Z score 0.01 0.02 
Class and teacher characteristics 

Mean prior ELA Z score of current peers 0.02 -0.02 

Teacher years of experience 11.49 11.85 
Grade level 

Grade 4 in 2016-2017 (%) 62.4 62.4 

Grade 5 in 2016-2017 (%) 37.6 37.6 
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Table 6.4 


Student Characteristics of Middle School Treatment and Comparison Groups After Coarsened Exact 
Matching, 2016-2017 School Year 


LDC sample: Secondary Comparison sample: Secondary 
Student characteristic (n = 3,214) (n = 3,214) 


Race/Ethnicity 


Hispanic (%) 56.1 57.1 
Black (%) 31.1 29.8 
Asian (%) 5.2 6.4 
White (%) 6.7 5.7 
Other (%) 0.9 0.9 
Female (%) 51.8 48.9 


Special programs status 


Poverty (%) 86.1 86.4 
English language learner (%) 12.8 13.5 
Special education (%) 23.6 25.8 
Gifted (%) 24.1 26.3 


Student prior achievement 
Mean prior year math Z score -0.40 -0.44 
Mean prior year ELA Z score -0.38 -0.37 


Class & teacher characteristics 


Mean prior ELA Z score of current peers -0.41 -0.37 

Teacher years of experience 8.7 8.4 
Grade level 

Grade 6 in 2016-2017 (%) 16.1 16.1 

Grade 7 in 2016-2017 (%) 41.1 41.1 

Grade 8 in 2016-2017 (%) 42.8 42.8 


6.3 Outcome Analysis Results: Elementary Sample 


As with middle school students, elementary students were typically associated with more 
than one core content teacher; the average treatment student was linked to 2.3 core content 
teachers and the average comparison student was linked to 2.6 teachers. For this reason, and 
to be consistent with the larger middle school analysis, we employed a multi-membership 
multiple classification (MMMC) design for the elementary analysis. Two different approaches 
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were used to model the LDC treatment intervention variable as a fixed effect at the student 
level. The first model was dosage dependent, which took into account some variation in 
elementary students’ level of exposure to LDC teachers. In this approach the treatment variable 
was structured as a continuous response variable between zero and one, based on exposure to 
LDC teachers for ELA, social studies/history, and/or science. In contrast, the second approach 
was modeled as dosage independent and considers any student exposed to any LDC 
intervention teacher via at least one subject as a treated individual. In this latter approach, the 
treatment variable was dichotomous, coded as one for LDC treated students and zero for 
comparison students. The outcome variable for both models was students’ 2017 New York 
State assessment scores in ELA. For technical reasons of evaluating effect sizes, New York State 
assessment scale scores were standardized to the study sample. 


Table 6.5 


2016-2017 LDC Elementary School Effect Estimates on NYS ELA 
Performance, Dosage-Dependent Model 


Variables Model coefficient (SE) 


LDC teacher treatment 0.072 (0.157) 


Student characteristics 


Hispanic -0.136 (0.160) 
Black -0.195 (0.150) 
Asian -0.141 (0.146) 
Other ethnicity -0.300 (0.208) 
Poverty -0.049 (0.065) 
Female 0.180 (0.052)* 
Grade 5 -0.119 (0.106) 
Teacher years of experience 0.000 (0.010) 
Mean prior ELA Z score of current peers 0.162 (0.093) 
Prior year math Z score 0.173 (0.044)* 
Prior year ELA Z score 0.543 (0.051)* 


Note. Since the average treatment student received a 0.724 treatment 
dosage we could estimate an average treatment on the treated (ATET) at 
0.724 * 0.072 = 0.052. This effect was not statistically significant. Inclusion 
rules led to removal of ELL students, students with disabilities, and gifted 
variables from the model. 

*9 = .05. 


The dosage-dependent effect of having an LDC teacher on elementary students’ ELA 
performance in 2016-2017 is shown in Table 6.5, and the dosage-independent effect is shown 
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in Table 6.6. Most treated students were only exposed to one teacher (the LDC intervention 
teacher); thus the average treated student in this sample received a 0.72 treatment dosage. 
Model results show no statistically discernible LDC effect on student outcomes. Similarly, the 
dosage-independent model revealed no statistically significant effect for LDC. In other words, 
both analyses revealed no evidence that students taught by LDC teachers scored dissimilarly on 
the ELA test to their matched peers in the comparison group. 


The effects of the covariates on student performance also were similar under both models 
and were in the expected directions. Prior ELA performance was the strongest predictor and 
prior math performance also helped explain the outcome. In addition to prior achievement, one 
demographic variable (gender) helped predict performance. Females performed at significantly 
higher levels than males. 


Table 6.6 


2016-2017 LDC Elementary School Effect Estimates on NYS ELA 
Performance, Dosage-Independent Model 


Variables Model coefficient (SE) 


LDC teacher treatment -0.066 (0.129) 


Student characteristics 


Hispanic -0.138 (0.160) 
Black -0.192 (0.150) 
Asian -0.143 (0.145) 
Other ethnicity -0.294 (0.207) 
Poverty -0.046 (0.065) 
Female 0.178 (0.052)* 
Grade 5 -0.124 (0.106) 
Teacher years of experience 0.000 (0.010) 


Mean prior ELA Z score of current 0.156 (0.092) 


peers 
Prior year math Z score 0.174 (0.044)* 
Prior year ELA Z score 0.542 (0.051)* 


Note. Inclusion rules led to removal of ELL, students with disabilities, and 
gifted variables from the model. 
*o = .05. 


6.4 Outcome Analysis Results: Middle School Sample 


Similar analyses were conducted at the middle school level, where the norm was student 
exposure to multiple teachers. Students could be exposed to anywhere between 0 and 17 
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different LDC teachers in the 2016-2017 school year. Using a multiple membership multiple 
classification (MMMC) design, each observation at Level 1 represented one student, which was 
linked to the ELA, social studies/history, and science teachers that the student was exposed to 
during the year. Weights across teachers for each student summed to a unity (1). 


As with the elementary sample, we present results of models that are both dosage 
dependent and dosage independent. The dosage-dependent effect of being exposed to LDC 
teachers in ELA, social studies, and science on middle school students’ ELA performance in 
2016-2017 is shown in Table 6.7, and the dosage-independent effect of having at least one LDC 
teacher is shown in Table 6.8. Model results show no statistically discernible LDC effect on 
student outcomes for either model. In other words, both analyses showed that students taught 
by LDC teachers scored similarly on the ELA test to their matched peers in the comparison 
group. 


The effects of the covariates on student performance were also similar in direction to 
those in the elementary school model, although due to the larger sample size more 
demographic variables were significant in the middle school analysis. Prior ELA and math 
performance, as well as ELL status, special education status, Black ethnicity, other ethnicity, and 
gender were significant predictors of ELA performance. In secondary schools, the average prior 
achievement of a student’s peers was also a significant predictor of ELA performance in the 
expected direction. 
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Table 6.7 


2016-2017 LDC Middle School Effect Estimates on NYS ELA 
Performance, Dosage-Dependent Model 


Variables Model coefficient 
(SE) 
LDC teacher treatment -0.017 (0.065) 


Student characteristics 


Hispanic -0.070 (0.036) 
Black -0.127 (0.038)* 
Asian 0.047 (0.045) 
Other ethnicity -0.170 (0.082)* 
Poverty -0.039 (0.022) 
Female 0.093 (0.015)* 
English language learner -0.101 (0.027)* 
Special education -0.258 (0.084)* 
Gifted 0.113 (0.083) 
Grade 7 0.045 (0.032) 
Grade 8 0.089 (0.035)* 
Teacher years of experience -0.004 (0.003) 
Total content weeks -0.003 (0.002) 


Mean prior ELA Z score of current 0.154 (0.029)* 


peers 
Prior year math Z score 0.179 (0.012)* 
Prior year ELA Z score 0.658 (0.013)* 


Note. Since the average treatment student received a 0.506 treatment 
dosage we could estimate an average treatment on the treated (ATET) at 
0.506 x -0.017 = -0.008. This effect was not statistically significant. 

KH 
p=.05. 
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Table 6.8 


2016-2017 LDC Middle School Effect Estimates on NYS ELA 
Performance, Dosage-Independent Model 


Variables Model coefficient (SE) 


LDC teacher treatment -0.019 (0.040) 


Student characteristics 


Hispanic -0.070 (0.036) 
Black -0.127 (0.038)* 
Asian 0.047 (0.045) 
Other ethnicity -0.170 (0.082)* 
Poverty -0.039 (0.022) 
Female 0.093 (0.015)* 
English language learner -0.101 (0.027)* 
Special education -0.258 (0.084)* 
Gifted 0.113 (0.083) 
Grade 7 0.045 (0.032) 
Grade 8 0.089 (0.035)* 
Teacher years of experience -0.004 (0.003) 
Total content weeks -0.003 (0.002) 
Mean prior ELA Z score of current peers 0.154 (0.029)* 
Prior year math Z score 0.179 (0.012)* 
Prior year ELA Z score 0.658 (0.013)* 
*9 = .05. 


6.5 Summary of Results 


Quasi-experimental design analyses of the impact of LDC on 2016-2017 student test 
scores in elementary and secondary schools produced two measures of impact effect at each 
school level. One impact effect measure took into account the dosage of LDC exposure while 
the other reflected any exposure to an LDC teacher and was thus dosage independent. Our 
2016-2017 analysis results did not reveal any statistically significant intervention effects. This 
finding should not be surprising given the early stage of intervention, with teachers having only 
completed one year of implementation. 
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7.0 Summary of Findings 


This annual report presents an initial look at LDC implementation in the first cohort of 29 
schools in the New York City Department of Education during their first year of implementation. 
We summarize these results organized by the three categories of evaluation questions we listed 
in Chapter 1 of this report. 


7.1 Program Characteristics and Implementation 


All participant groups in this district were generally positive about LDC characteristics and 
implementation at their sites. Questions about program quality and impact received positive 
responses from a majority of respondents. A quarter of survey respondents took the time to 
specify in open-ended responses that they found all LDC supports helpful. 


Teachers valued the collaborative nature of the initiative, as evidenced by their 
participation in regular PLC meetings and informal discussions. A large majority of teachers 
credited LDC with making them more likely to collaborate with other teachers, not only in their 
grade levels and content areas but outside of them as well. In addition to peer feedback, 
teachers valued coach feedback greatly. Positive ratings of LDC coaches were nearly universal. 


Online course material was revised sufficiently following pilot teachers’ feedback that it 
became a strength of the program this year. Most teachers rated the online courses as highly 
useful, helpful, and relevant. A small number of users found the courses difficult to navigate 
and read, redundant, and not specific enough to be helpful. 


There were 184 teachers over multiple content areas across all grade levels, and some of 
these teachers did not feel well supported by the available resources, particularly in science and 
math. 


Module quality suffered when teachers simply adapted existing modules without making 
significant changes or providing insight about their reasoning for instructional decisions. 
Although the module building platform provides optional areas where teachers provide such 
reasoning and reflection after the fact, most teachers did not utilize these areas. In addition, 
few teachers provided student work. Without either student work or teacher reflection, it was 
difficult for raters to evaluate whether or not instructional goals were met. Teachers who 
implemented modules and provided evidence of the implementation engaged more deeply 
with the LDC platform. 


7.2 Contextual Factors and Implementation 


The people who supported teachers in the LDC implementation—project liaisons, 
administrators, LDC coaches, and LDC staff—received extremely positive responses for 
helpfulness, timeliness, and level of knowledge. Questions about the quality of support that 
participants received often had universal or nearly universal agreement. The effectiveness of 
the people involved was the most notable facilitating factor in implementation. 


71 


The most common suggestion to improve implementation was to provide more time, 
which included more time and support to both plan and implement modules, not necessarily 
during scheduled PLC meetings. Other suggestions included starting earlier in the school year or 
during the summer. 


Related to time was the problem of technology. This issue had two points. First, the 
logistics of Zoom meetings were difficult at some sites. This was perceived as a frustrating 
waste of time. These open-ended responses stood in contrast to the finding that most teachers 
thought their school’s technology was adequate for LDC implementation. This indicates that 
although the Zoom meetings worked for the majority of participants, failed meetings resulted 
in highly negative perceptions for those affected. The second problem with technology was that 
some teachers needed far less time and support to understand CoreTools and other teachers 
need far more. In general, however, the complexity of CoreTools and the need to use multiple 
sites like The Teaching Channel was a significant barrier for those teachers who were affected. 
Thus, the most effective use of limited teacher and coach time during PLC meetings were 
constrained by the different technology needs of the sites and of the participating teachers. 


The issue of differentiation among teachers touches on many factors in addition to 
technology, such as content area as described above, and as the initiative progresses, the level 
of experience and knowledge about the LDC model. As the pool of teachers both deepens and 
widens, the ability of the program to differentiate will likely affect participants’ experience to a 
great extent. 


7.3 Program Impacts 


Based on survey results, LDC was perceived to have positive impacts on both teacher 
practice and student learning. Administrators were particularly enthusiastic, with a large 
majority believing that LDC helped students in multiple facets of literacy and skills acquisition. 
Teachers were slightly more cautious but still positive. 


Teacher self-reports of change in their own ability over the year indicated that they felt 
they had improved most in areas related to instructional planning, which aligned with the 
primary content of LDC sessions. Slightly fewer teachers reported positive change in areas 
involving collecting and using student data to inform instruction, which again corresponded to 
the lower number of teachers who implemented modules. 


Module analysis showed teacher strengths in creating an effective writing task and 
aligning to standards. 


Our preliminary analysis of the 2016-2017 student data did not reveal any statistically 
significant intervention effects. This finding should not be surprising given the early stage of 
intervention, with teachers having only completed one year of implementation while LDC 
implementation is a two-year process. 
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Appendix A: 
LDC Module Rating Dimensions 


Each module was rated on six dimensions. All of these were rated using a 5-point scale. With the first 
five, anchor points ranged from not present or realized to fully present or realized. In contrast, the 
overall dimension ranged from inadequate to advanced. 


Dimension 


1 Effective Writing Task 


Definition 

Degree to which teaching task makes effective use of the template task’s writing mode (i.e., argumentation 
or explanation); requires sustained writing and effective use of ideas and evidence to substantiate claims; 
and is feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
e Evaluate the difficulty or ease students may encounter trying to answer the question. 


e Compare module teaching task to teaching task template options. 


Anchor Points Description 


5 Fully Present or Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order 
thinking and writing, and are appropriate for the grade-level and 
subject matter. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized | Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level 
and subject matter. 


2 Barely Present or Realized 


1 Not Present or Realized There is minimal evidence of an effort to identify an explicit and 
clear teaching task and performance expectations that provide 
opportunities for students to think critically and are appropriate for 
the grade-level and subject matter. 
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Dimension 


2 


Alignment to CCSS, Local, State Literacy and Content Standards 


Definition 
Extent to which module addresses content essential to the discipline, as well as reading comprehension 


and writing standards informed by local and state standards. 


Main Sources of Information 


Module Creator Handout (Task) 


Read and evaluate the standards included in the module. 


Module should include ELA as well as subject matter CCSS/state standards. 


Compare and contrast the standards the module includes with those that could have been included. 


Particular attention to content standards (CCSS History/Social Studies, Science, and Technical 
Subjects); State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


Anchor Points 


Description 


5 


Fully Present or Realized 


Sufficiently Present or Realized 


Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the 
topic and teaching task. 


Moderately Present or Realized 


Barely Present or Realized 


Module broadly addresses content essential to CCSS and local or 
state standards in science or social studies and reading 
comprehension and writing. Standards are sufficiently aligned to the 
topic and teaching task. 


Not Present or Realized 


Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 
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Dimension 


3 


Fidelity to LDC Module Instruction 


Definition 


Degree to which module instruction, activities, and teaching task address each of the four stages of 
instructional practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information 


Module Creator Handout (Instruction) and Information Sheet 


e Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


Anchor Points 


Description 


5 Fully Present or Realized 


The module instruction, activities, and teaching task reflect 
deliberate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect demonstrable effort 
to develop instructional scaffolding within and across each stage of 
instruction. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


The module instruction, activities, and teaching task reflect 
moderate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect sufficient effort to 
develop instructional scaffolding within and across each stage of 
instruction. 


2 Barely Present or Realized 


1 Not Present or Realized 


The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect inadequate effort to 
develop instructional scaffolding within and across each stage of 
instruction. 
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PyTnat=tarsiteyal 


A 


Quality Instructional Strategies 


Definition 


Degree to which the module provides clear instructional strategies aimed at helping students develop 


literacy skills and successfully complete the teaching task. In addition, the degree to which module 


instruction and activities scaffold critical thinking and performance in a way that is meaningful within the 


context of a given field or subject matter. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


e Evaluate extent to which the module activities scaffold critical thinking and student performance within 


the context of the subject matter at the core of the teaching task. 


e Evaluate extent to which instructional strategies guide student learning in literacy and ability to 


complete the teaching task. 


Anchor Points 


Description 


5 Fully Present or Realized 


Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking 
in social studies or science. There is explicit attention to helping 
students develop an accurate understanding of the topic and 
teaching task, and literacy skills necessary to complete the writing 
task successfully. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Instructional strategies and activities are available to support 
adequate student learning and critical thinking in social studies or 
science. There is moderate attention to helping students develop an 
understanding of the topic and teaching task, and literacy skills 
necessary to complete the writing task. 


2 Barely Present or Realized 


1 Not Present or Realized 


Limited instructional strategies and activities are available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding 
of the topic and teaching task, or literacy skills necessary to 
complete the writing task. 
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ts) 


Coherence and Clarity of Module 


Definition 


The degree of logical alignment found between the teaching task and the goals of the module with the 


readings, mini-tasks, and instructional strategies. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


Anchor Points 


Description 


5 Fully Present or Realized 


Strong alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Moderate alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


2 Barely Present or Realized 


1 Not Present or Realized 


Poor alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 
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6 Overall Impression 


Definition 


A holistic assessment of the LDC Module. 


Main Sources of Information 


Module Creator Handout, Classroom Handouts, and Student Work 


e To what extent does this module contribute to student college readiness and development of advanced 
literacy skills? 


Anchor Points 


5 Advanced LDC Module Implementation 


Proficient LDC Module Implementation 


4 
3 Adequate LDC Module Implementation 
2 


Marginal LDC Module Implementation 


1 Inadequate LDC Module Implementation 
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Appendix B: 
Teacher Survey and Responses 


2016-2017 LDC Teacher Survey: NYCDOE 


LDC Participation 


T1. Please select your school from the drop down box. 
Teachers are skipped to T3 if they teach in an elementary school. 


T2a. In the current school year (2016-17), how many classes did you teach? 


| | classes N-=169, Mean = 3.67, Range: 1-15 


T2b. In how many of these classes did you use LDC modules and/or mini-tasks? 


| | classes N=169, Mean = 2.35, Range: 0-5 


T2c. In what content areas did you use LDC modules and/or mini-tasks? 


T2d. In what grades did you use LDC modules and/or mini-tasks? 


T3. Prior to the current school year (2016-17), did you have any experience with LDC? 


(N = 184) 
[ ] Yes 19 teachers (10.3%) 
No y Skip to T5a 165 teachers (89.7%) 
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T4. How many of the following did you teach prior to the current school year (2016-17)? 


| | LDC modules N = 19, Mean = 1.21, Range: 0-6 


| | LDC mini-tasks, outside of modules N=19, Mean =1.21, Range: 0-8 


Professional Learning Community and Teacher Collaboration 


T5a. Did you participate this year in a Professional Learning Community (PLC) at least partly 
focused on implementing LDC in your school? 


(N = 159) 
Yes y Skip to T6 147 teachers (92.5%) 
No 12 teachers (7.5%) 


T5b. Did you use any LDC tools in your instructional planning or classroom instruction this 


year? 
(N = 12) 
Yes <b Skip to 5d 9 teachers (75.0%) 
| ] No 3 teachers (25.0%) 


T5c. Why did you choose not to use any LDC tools in your instructional planning or 
classroom instruction this year? 


[Survey ends here for respondents answering question T5c] 
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T5d. What LDC tools did you use during the current school year? Select all that apply. 


CoreTools online platform to access existing modules or mini-tasks 


CoreTools online platform to design modules or mini-tasks 


LDC online courses 


Modules or mini-tasks given to me by other teachers in my school 


Other (please specify) 


(N= 9) 

5 teachers (55.6%) 
7 teachers (77.8%) 
1 teachers (11.1%) 
3 teachers (33.3%) 
0 teachers (0%) 


[Survey ends here for respondents answering question T5d] 


T6. About how often did your LDC PLC meet? 


Less than once a month 
Once a month 
Every other week 


Once a week 


y Skip to T8 
y Skip to T8 


Twice a week or more often 


(N =172) 

1 teachers (0.6%) 

15 teachers (8.7%) 
45 teachers (26.2%) 
100 teachers (58.1%) 
11 teachers (6.4%) 


T7. What were the primary barriers preventing your LDC PLC from meeting weekly? Select 
all that apply. 


PLC time was not protected. 


PLC members had limited interest in attending meetings. 
School administrator did not make it a priority. 


Project liaison did not provide sufficient organizational support. 


Not enough teachers participated. 


PLC members had other priorities that compete with LDC participation. 


Other (please specify) 
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(N = 61) 

11 teachers (18.0%) 
'5 teachers (8.2%) 

7 teachers (11.5%) 
1 teachers (1.6%) 

3 teachers (4.9%) 
35 teachers (57.4%) 
19 teachers (31.1%) 


T8. About how often did you have informal discussions (as opposed to scheduled meetings) 


about LDC with teachers in your LDC PLC? 
(N = 172) 


Less than once a month 32 teachers (18.6%) 
Once a month 18 teachers (10.5%) 
Every other week 37 teachers (21.5%) 
Once a week 50 teachers (29.1%) 
Twice a week or more 35 teachers (20.3%) 


T9. On average, how long did your school’s LDC PLC meetings typically last? 


(N = 172) 
Less than 45 minutes 29 teachers (16.9%) 
45 to 59 minutes 92 teachers (53.5%) 
[| | 60 to 74 minutes 32 teachers (18.6%) 
75 minutes or more 19 teachers (11.0%) 


LDC Training and Support 


T10. How effective was your LDC PLC in the following areas? 


A little 
effective 


Moderately 
effective 


Very 
effective 


38 (22.4%) 


65 (38.2%) 


AS (26.5%) 


24 (14.1%) 


65 (38.2%) 


70 (41.2%) 


36 (21.2%) 


67 (39.4%) 


59 (34.7%) 


26 (15.3%) 


77 (45.3%) 


54 (31.8%) 


Not 

effective 
Demonstrating the usefulness of LDC for ‘ 
teachers (N = 170) 22 (12.9%) 
Creating an environment in which teachers were 11 (6.5%) 
comfortable working together (N = 170) 
Fostering an environment where teachers 
shared their instructional plans with colleagues 8 (4.7%) 
(N = 170) 
Allowing space for shared problem solving é 
(N = 170) 13 (7.6%) 
Helping teachers learn to develop modules és 
(N = 170) 10 (5.9%) 


32 (18.8%) 


73 (42.9%) 


55 (32.4%) 
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T11. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 
Clarity of information presented (N = 170) 11 (6.5%) | 44(25.9%) | 80(47.1%) | 35 (20.6%) 
Relevance of information presented (N = 170) 16 (9.4%) | 39 (22.9%) | 70(41.2%) | 45 (26.5%) 


Ease of use (N = 170) 


22 (12.9%) 


45 (26.5%) 


75 (44.1%) 


28 (16.5%) 


Usefulness of resource documents (e.g., LDC 
Pitfall Checklist, CCSS Mental Markers, etc.) 
(N = 170) 


13 (7.6%) 


52 (30.6%) 


73 (42.9%) 


32 (18.8%) 


Usefulness of videos (N = 170) 


17 (10.0%) 


56 (32.9%) 


77 (45.3%) 


20 (11.8%) 


Degree to which course material helped me to 
create and/or adapt LDC modules (N = 170) 


19 (11.2%) 


46 (27.1%) 


77 (45.3%) 


28 (16.5%) 


Opportunity to extend learning when needed or 
desired (N = 170) 


18 (10.6%) 


47 (27.6%) 


65 (38.2%) 


40 (23.5%) 


T12a. Overall, were you able to get the feedback and support you needed from your LDC coach 
(through written feedback in LDC CoreTools, or coaching and modeling in your LDC PLCs) to 
plan, teach, reflect on, and revise LDC modules)? 


(N = 170) 


Yes 161 teachers (94.7%) 
No 9 teachers (5.3%) 


T12b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(N = 170) 


Yes 161 teachers (94.7%) 


No 9 teachers (5.3%) 
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T13. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not 
nee Not A little Moderatel Very 
helpful helpful y helpful helpful 
nee inte | ae Bae a3 oA 
the teacher work rubric) (N = 170) (7.6%) (3.5%) (20.0%) (31.2%) (37.6%) 
One-on-one Zoom video conference and/or call 25 10 30 39 64 
with your LDC coach (N = 168) (14.9%) (6.0%) (17.9%) (23.2%) (38.1%) 
Coach-facilitated discussions on the Teaching 38 7 32 42 48 
Channel Teams platform (N = 167) (22.8%) (4.2%) (19.2%) (25.1%) (28.7%) 
Email or phone communication with your LDC 50 5 25 40 50 
coach (N = 170) (29.4%) (2.9%) (14.7%) (23.5%) (29.4%) 
Other (please specify) (N= 40) 21 2 Z 5 10 
(52.5%) (5.0%) (5.0%) (12.5%) (25.0%) 


Module Creation 


714. During the current school year (2016-17), how many LDC modules did you individually 
or collaboratively adapt from existing modules (e.g., modules you created in a prior 
year and/or modules found in the LDC Library in CoreTools)? 


| | Adapted modules 


N = 145, Mean = 1.85, Range: 0-20 


T15. During the current school year (2016-17), how many LDC modules did you create, 
either individually or with colleague(s)? Only include modules built from scratch, not 


those adapted from existing modules in the LDC library. 


| | New modules 
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N = 145, Mean = 1.57, Range: 0-5 


T16. How did members of your PLC collaborate to create LDC modules? 


Check all that apply. 


Other (please specify) 


Modules were created by individual teachers. 


Modules were created by the PLC as a whole. 


Modules were created by teams of two or more teachers. 


(N = 185) 


112 teachers (60.5%) 


117 teachers (63.2%) 


20 teachers (10.8%) 


3 teachers (1.6%) 


T17. Please indicate to what extent you were able to do each of the following when creating 


LDC modules and/or mini-tasks. 


(N = 170) 


A 
Notatall | Alittle bit | moderate Aaree 
extent 
extent 
Select a set of focus standards for a writing assignment di 24 59 86 
(N = 170) (0.6%) (14.1%) (34.7%) (50.6%) 
Create a standards-driven writing assignment task 3 25 52 90 
(N = 170) (1.8%) (14.7%) (30.6%) (52.9%) 
Select high quality, complex texts and other materials 10 32 66 62 
that allowed students to engage in deeper learning 
3 (5.9%) (18.8%) (38.8%) (36.5%) 
(N = 170) 
Create a writing assignment that provided multiple 8 7 71 64 
apes eee for students to engage with the material (4.7%) (15.9%) (41.8%) (37.6%) 
(N = 170) 
Identify the skills students need to develop in order to 7 24 71 68 
complete a writing assignment (N = 170) (4.1%) (14.1%) | (41.8%) | (40.0%) 
Create daily lessons to teach the skills a student needs to 14 39 69 48 
complete a writing assignment (N = 170) (8.2%) (22.9%) (40.6%) | (28.2%) 
Plan for a variety of methods to assess student progress a2 47 71 40 
(e.g., mini-task scoring guides) (N = 170) (7.1%) (27.6%) (41.8%) (23.5%) 
ee | ee ee 
oer (7.6%) | (21.2%) | (40.6%) | (30.6%) 
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Classroom Implementation 


T18. How many total LDC modules did you teach during the current school year (2016-17)? 


| | Modules N = 170, Mean = 2.36, Range: 0-8 


T19. Outside of modules, approximately how many individual LDC mini-tasks did you teach during 
the current school year (2016-17)? 


a Mini-tasks N = 170, Mean = 3.56, Range: 0-35 


T20. Please indicate to what extent you were able to do each of the following activities when 
teaching LDC modules and/or mini-tasks. 


A 
Notatall | Alittlebit | moderate | “8reat 
extent 
extent 
Engage students in productive struggle through P ss a m 


challenging texts, discussion questions, and other 


instructional supports (N = 170) (2.4%) (12.9%) (51.2%) (33.5%) 


Provide support for students to complete the 7 29 76 58 
assignment by differentiating instruction (N = 170) (4.1%) (17.1%) (44.7%) (34.1%) 
Systematically collect information about student 8 38 78 46 
progress (N = 170) (4.7%) (22.4%) (45.9%) (27.1%) 


Provide feedback to students using shared 
expectations for learning (e.g., a mini-task scoring 
guide) (N = 170) 


7 34 76 53 
(4.1%) | (20.0%) | (44.7%) (31.2%) 


Locate evidence of performance of standards in 5 34 74 57 
THC ERE Wore een (2.9%) | (20.0%) | (43.5%) | (33.5%) 
Use evidence of student performance of standards to 9 31 74 te 
modify instructional plans (N = 170) (5.3%) (18.2%) (43.5%) (32.9%) 


T21. Did you teach a One-Text One-Week module? 


(N = 170) 
Yes 86 teachers (50.6%) 
No ib Skip to T24 84 teachers (49.4%) 


T22. What was the name of the One-Text One-Week module? 
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T23. 


T24. 


T25. 


T26. 


Which of these statements best describes how you created the One-Text One-Week module? 


(N = 86) 
| created a module from a template in CoreTools. 62 teachers (72.1%) 
| copied and adapted another teacher’s module 24 teachers (27.9%) 


from the LDC Library in CoreTools. 


What module did you work on most heavily during the current school year (2016-17)? This 
module is typically one you worked on after the One-Text One-Week module. 


Which of these statements best describes how you created the module named in the previous 
question? 


(N = 170) 
| created a module from a template in CoreTools. 113 teachers (66.5%) 
| copied and adapted another teacher’s module 57 teachers (33.5%) 
from the LDC Library in CoreTools. 
Did you teach this module in your classroom? 
(N = 170) 
Yes, | have already taught this module this year. 140 teachers (82.4%) 
No, but | plan to teach this module before the end of the 2016-17 school 14 teachers (8.2%) 
year. 
No, but | plan to teach this module during next school year. 10 teachers (5.9%) 


No. | do not currently have plans to teach this module in my classroom. 6 teachers (3.5%) 
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Module Peer Review 


T27. How many modules did you submit online for LDC National Peer Review during the 
current school year (2016-17)? 


| | Modules if none, Lb skip to T29 N = 170, Mean = 0.82, Range: 0-5 


T28. How helpful did you find the National Peer Review process in improving the quality of 
your module? 


(N = 82) 
Not helpful 17 teachers (20.7%) 
A little helpful 26 teachers (31.7%) 
|__| Moderately helpful 30 teachers (36.6%) 
| | Very helpful 9 teachers (11.0%) 
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7 


Impact on Teacher Practice and Learning 


T29. Between the beginning and end of this year’s work with LDC, please indicate how 


much your ability in the following areas has improved: 


Not at all A little Moderately me ereat 
deal 
Selecting a set of focus standards for a writing assignment 15 af 63 54 
(N = 169) (8.9%) (21.9%) (37.3%) (32.0%) 
Creating standards-driven writing tasks of sufficient 15 36 67 51 
quality and scope to allow students to think deeply about 3.9% 21.3% 9.6% 30.2% 
disciplinary content and practice literacy skills (N = 169) (8.9%) (21.3%) (39.6%) (30.2%) 
Identifying the skills students need to develop in order to 13 42 68 46 
complete a writing assignment (N = 169) (7.7%) (24.9%) (40.2%) (27.2%) 
Creating daily lessons to teach each skill a student needs io 39 68 40 
to complete a writing assignment (N = 169) (13.0%) (23.1%) (40.2%) (23.7%) 
Assessing students’ progress as they work toward 23 36 66 44 
completing a writing assignment (N = 169) (13.6%) (21.3%) (39.1%) (26.0%) 
Tracking and analyzing evidence about student progress 26 29 61 43 
in a systematic way (N = 169) (15.4%) (23.1%) (36.1%) (25.4%) 
Identifying patterns of student understandings or 24 38 63 44 
misconceptions (N = 169) (14.2%) (22.5%) (37.3%) (26.0%) 
Using evidence of student performance of standards to 23 34 62 50 
shape future instructional decisions (N = 169) (13.6%) (20.1%) (36.7%) (29.6%) 
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T30. Please indicate the degree to which you agree or disagree with the statements below. 


SHane Disagree Agree Stenely 

disagree agree 
Participating in LDC raised my expectations for students’ 13 38 80 38 
writing. (N = 169) (7.7%) (22.5%) (47.3%) (22.5%) 
Using LDC modules became an important part of my 22 43 81 23 
instructional practice. (N = 169) (13.0%) (25.4%) (47.9%) (13.6%) 
Implementing LDC helped me incorporate my state’s 16 42 79 32 


College- and Career-Ready Standards into my instruction. (9.5%) (24.9%) (46.7%) (18.9%) 


(N = 169) 
LDC helped me incorporate writing assignments into my 20 30 82 a7 
existing curriculum. (N = 169) (11.8%) (17.8%) (48.5%) (21.9%) 


lam more likely to collaborate with other teachers on 
designing instruction after participating in our LDC 
Professional Learning Community. (N = 169) 


19 eal 84 35 
(11.2%) | (18.3%) | (49.7%) | (20.7%) 


Participating in LDC helped me develop working 
relationships with teachers in different grades and/or 
subjects. (N = 169) 


21 27 90 31 
(12.4%) | (16.0%) | (53.3%) | (18.3%) 


| often incorporated LDC mini-tasks into my non-LDC 19 53 rail 26 
instruction. (N = 169) (11.2%) (31.4%) (42.0%) (15.4%) 
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Impact on Student Learning 


T31. Please indicate the degree to which you agree or disagree with the statements below. 


SHOuBly Disagree Agree stronely 

disagree agree 
LDC helped students develop reading skills. LS 30 80 ue 
(N = 144) (10.4%) (20.8%) (55.6%) (13.2%) 
LDC was effective in improving students’ content 14 Ball 76 23 
knowledge. (N = 144) (9.7%) (21.5%) (52.8%) (16.0%) 
LDC modules effectively supported students in 11 26 82 25 
completing writing assignments. (N = 144) (7.6%) (18.1%) (56.9%) (17.4%) 
LDC was effective in improving the quality of students’ 14 31 74 25 
writing. (N = 144) (9.7%) (21.5%) (51.4%) (17.4%) 
LDC supported students’ development of skills needed for 15 25 82 Ze 
college and career readiness. (N = 144) (10.4%) (17.4%) (56.9%) (15.3%) 
ie cre cae te ees | ee - ie . 

e P eens "| (7.6%) | (18.1%) | (54.2%) | (20.1%) 

(N = 144) 
LDC helped students develop speaking and listening skills. 14 37 73 20 
(N = 144) (9.7%) (25.7%) (50.7%) (13.9%) 
Overall, LDC helped improve students’ literacy 14 30 77 23 
performance (N = 144) (9.7%) (20.8%) (53.5%) (16.0%) 
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Project Liaison Support 


The following question refers to the LDC project liaison in your school. This is the teacher or administrator 


leading your Professional Learning Community work. 


T32. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly 


Strongly : 

disagree DBSETEE Agree agree 
Our school’s LDC project liaison effectively supported our 3 8 61 val 
Professional Learning Community meetings. (N = 143) (2.1%) (5.6%) | (42.7%) | (49.7%) 
When | had questions about LDC, | felt comfortable 3 7 63 70 
approaching our school’s project liaison. (N = 143) (2.1%) (4.9%) | (44.1%) | (49.0%) 

: af 4 13 63 63 

Our project liaison had a strong grasp of LDC. (N = 143) (2.8%) (9.1%) (44.1%) (44.1%) 
Our project liaison helped teachers align LDC to broader 6 16 60 61 
school instructional goals. (N = 143) (4.2%) (11.2%) | (42.0%) | (42.7%) 
Our project liaison offered useful feedback for the design 7 13 61 62 
and revision of LDC modules. (N = 143) (4.9%) (9.1%) | (42.7%) | (43.4%) 
Our project liaison was effective in inviting teachers to join 4 10 64 65 
the LDC initiative. (N = 143) (2.8%) (7.0%) | (44.8%) | (45.5%) 


Ke) School Administrator Support 


T33. What proportion of PLC meetings focused on LDC did your school administrator attend? 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 
About one half of LDC PLCs 
About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 
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(N = 168) 


70 teachers (41.7%) 
30 teachers (17.9%) 
25 teachers (14.9%) 
11 teachers (6.5%) 

32 teachers (19.0%) 


The following questions refer to the school administrator who oversees the LDC project at your school. 


T34. How many times did your school administrator observe you teach an LDC mini-task 
during the current school year (2016-17)? 


(N = 168) 


0 times 74 teachers (44.0%) 
1time 53 teachers (31.5%) 
2 times 20 teachers (11.9%) 
3 or more times 21 teachers (12.5%) 


T35. Please indicate the degree to which you agree or disagree with the following statements. 


My school administrator... Strongly Disagree Agree strongly 
disagree agree 
. . _ a 36 7 24 
had a firm understanding of LDC. (N = 168) (6.5%) (21.4%) | (57.7%) (14.3%) 
protected common planning time for our LDC 5 18 93 SZ 
Professional Learning Community to meet. (N = 168) (3.0%) | (10.7%) | (55.4%) (31.0%) 
encouraged teachers to participate in LDC. (N = 168) (2 ~ (4 - rr en 
. 0 . 0 . 0 . 0 
expressed concerns that implementing LDC is taking 29 69 52 18 
time away from other instructional priorities. ‘ ‘ ‘ 
(N = 168) (17.3%) | (41.1%) | (31.0%) (10.7%) 
communicated how using LDC’s tools supported specific 10 31 o7 10) 
school initiatives and/or goals. (N = 168) (6.0%) (18.5%) | (57.7%) (17.9%) 
provided me with feedback about my LDC planning 20 50 80 18 
and/or instruction. (N = 168) (11.9%) | (29.8%) | (47.6%) (10.7%) 
made formative assessment a priority at my school. 2 1g 107 40 
(N = 168) (1.2%) (11.3%) | (63.7%) (23.8%) 
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si) Teacher Leadership Role 


T36. Please indicate the degree to which you agree or disagree with the following 
statements about your role in your school’s LDC implementation. 


stransly Disagree Agree Strongly 
disagree agree 
| was involved in setting instructional goals for the LDC work 1S 40 68 20 
at my school. (N = 143) (10.5%) | (28.0%) | (47.6%) | (14.0%) 
| was involved in discussions about how to expand LDC 24 51 49 19 
implementation at my school in future years. (N = 143) (16.8%) | (35.7%) | (34.3%) | (13.3%) 
| had the opportunity to work with our LDC project liaison and 1 38 63 1 


our administrator to help shape LDC implementation. (14.7%) (26.6%) | (44.1%) | (14.7%) 


(N = 143) 
LDC helped me create writing assignments to use within my 12 20 77 34 
current curriculum. (N = 143) (8.4%) (14.0%) | (53.8%) | (23.8%) 


lam interested in learning more about how to lead LDC 
implementation at my school by facilitating with the virtual 
coach, providing feedback to my peers, etc. (N = 143) 


26 24 65 28 
(18.2%) | (16.8%) | (45.5%) | (19.6%) 


12 Facilitators and Barriers 


T37. Please indicate the degree to which you agree or disagree with the following statements. 


Sonely Disagree Agree strangely 
disagree agree 
5 35 82 41 
My LDC PLC i fficient ti t t. (N = 167 
y was given sufficient time to meet. ( ) (5.4%) (21.0%) | (49.1%) (24.6%) 
| felt adequately prepared to effectively implement LDC 13 35 88 31. 
modules in my classroom. (N = 167) (7.8%) (21.0%) | (52.7%) | (18.6%) 
It was challenging to find content-rich reading materials for 14 67 65 21 
the LDC modules | developed. (N = 167) (8.4%) (40.1%) | (38.9%) | (12.6%) 
My school had adequate technology to support teachers’ use =] 26 88 44 
of LDC. (N=167) (5.4%) (15.6%) | (52.7%) (26.3%) 
It was easy to find and adapt LDC mini-tasks for use in my 16 33 86 32 
classroom. (N = 167) (9.6%) (19.8%) | (51.5%) | (19.2%) 
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Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 


O Zoom meetings, written feedback on teacher work in LDC CoreTools, Teaching Channel 
discussions, emails, etc. 


e  /n-person coaching 


O Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


T38. What supports did you find the most useful and why? 


T39. What supports were not helpful and why? 


T40. In what ways could LDC implementation be improved in your school in the future? 
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Appendix C: 
Project Liaison Survey and Responses 


2016-2017 LDC Project Liaison Survey: NYCDOE 


i) LDC Participation 


PL1. Prior to the current school year (2016-17), did you have any experience with LDC? 


(N = 30) 
Yes 4 liaisons (13.3%) 
| ] No <5 Skip to PL3 26 liaisons (86.7%) 


PL2. How many of the following did you teach prior to the current school year (2016-17)? 


| | LDC modules N = 4, Mean = 2.5, Range: 0-6 


| | LDC mini-tasks, outside of modules N = 4, Mean = 2.5, Range: 0-8 


Professional Learning Community and Teacher Collaboration 


The following questions involve the LDC Professional Learning Community (PLC) that you are leading. 


PL3. About how often did your LDC PLC meet? 


(N = 30) 
Less than once a month 0 liaisons (0%) 
Once a month 3 liaisons (10.0%) 
Every other week 5 liaisons (16.7%) 
Once a week y Skip to PLS 21 liaisons (70.0%) 
| ] Twice a week or more often y Skip to PLS 1 liaisons (3.3%) 
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PL4. What were the primary barriers preventing your LDC PLC from meeting weekly? 
Select all that apply. 


(N = 30) 

PLC time was not protected. 2 liaisons (6.7%) 
PLC members had limited interest in attending meetings. 0 liaisons (0%) 
School administrator did not make it a priority. 0 liaisons (0%) 

| was unable to provide sufficient organizational support. 0 liaisons (0%) 
Not enough teachers participated. 0 liaisons (0%) 
PLC members had other priorities that competed with LDC participation. 0 liaisons (0%) 
Other (please specify) 4 liaisons (13.3%) 


PL5. About how often did you have informal discussions (as opposed to scheduled 
meetings) about LDC with teachers in your LDC PLC? 


(N = 30) 
Less than once a month 1 liaisons (3.3%) 
Once a month 4 liaisons (13.3%) 
Every other week 11 liaisons (36.7%) 
| ] Once a week 6 liaisons (20.0%) 
Twice a week or more 8 liaisons (26.7%) 


PL6. On average how long did your school’s LDC PLC meetings typically last? 


(N = 30) 
Less than 45 minutes 8 liaisons (26.7%) 
45 to 59 minutes 11 liaisons (36.7%) 
60 to 74 minutes 8 liaisons (26.7%) 
75 minutes or more 3 liaisons (10.0%) 
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LDC Training and Support 


PL7. How effective was your LDC PLC in the following areas? 


Not Alittle Moderately Very 

effective effective effective effective 
oo the usefulness of LDC for teachers 2 (6.7%) 8 (26.7%) 12 (40.0%) 8 (26.7%) 
Creating an environment in which teachers are ‘ és ‘ 5 
comfortable working together (N = 30) 1 (3.3%) 1 (3.3%) TA (AGIA) | 14(46,7%) 
Fostering an environment where teachers share Ft ‘ A ‘ 
their instructional plans with colleagues (N = 30) 1 (3.3%) 4 (13.3%) 14 (46.7%) | 11 (36.7%) 
fe space for shared problem solving 0 (0%) 3 (10.0%) 19 (63.3%) 8 (26.7%) 
Helping teachers learn to develop modules 1 (3.3%) 2 (6.7%) 17 (56.7%) | 10 (33.3%) 


(N = 30) 


PL8. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


or desired (N = 30) 


Poor Fair Good Excellent 

Clarity of information presented (N = 30) 0 (0%) 4 (13.3%) | 16(53.3%) | 10 (33.3%) 
Relevance of information presented (N = 30) 0 (0%) 4 (13.3%) | 14(46.7%) | 12 (40.0%) 
Ease of use (N = 30) 3 (10.0%) | 5(16.7%) | 16 (53.3%) 6 (20.0%) 
Usefulness of resource documents (e.g., LDC Pitfall : ‘5 ‘ ‘ 

Checklist, CCSS Mental Markers, etc.) (N = 30) 2 (6.7%) 6 (20.0%) 14 (80.7%) 8 (26.7%) 
Usefulness of videos (N = 30) 1 (3.3%) 7 (23.3%) | 18(60.0%) | 4 (13.3%) 
Degree to which course material helped teachers ‘ : ‘ ‘ 

to create and/or adapt LDC modules (N = 30) 1 (3.3%) 5 (16.7%) 17 (56.7%) 7 (23.3%) 
Opportunity to extend learning when needed 0 (0%) 8 (26.7%) 13 (43.3%) 9 (30.0%) 
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PL9a. Overall, were you able to get the feedback and support you needed from your LDC coach 
(through written feedback in LDC CoreTools, or coaching and modeling in your LDC PLCs) to 
plan, teach, reflect on, and revise LDC modules? 

(N = 30) 
Yes 29 liaisons (96.7%) 


No 1 liaisons (3.3%) 


PL9b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 
timely manner? 
(N = 30) 
Yes 29 liaisons (96.7%) 
No 1 liaisons (3.3%) 


PL10. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not : 
use Not A little Moderately Very 

helpful helpful helpful helpful 

ee nn ee! ao 0 3 2 
the teacher work rubric) (N = 30) (3.3%) (0%) (10.0%) (33.3%) (53.3%) 

One-on-one Zoom video conference and/or 1 0 5 9 15 
call with your LDC coach (N = 30) (3.3%) (0%) (16.7%) (30.0%) (50.0%) 

Coach-facilitated discussions on the Teaching 2 0 8 11 5 
Channel Teams platform (N = 30) (6.7%) (0%) (26.7%) (36.7%) (30.0%) 

Email or phone communication with your LDC Hl 0 4 | 16 
coach (N = 30) (3.3%) (0%) (13.3%) (30.0%) (53.3%) 

1 0 0 1 1 

Oth | ify) (N=3 

EPipease SPEC eae (33.3%) (0%) (0%) (33.3%) | (33.3%) 
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PL11. How many in-person and/or online LDC professional development offerings for school 
administrators and project liaisons did you attend during the current school year (e.g., 


Launch Day [in person], spring Project Liaison Meeting [in person], Bi-Monthly Community 
Meetings [online], Bi-Monthly Coach Meetings [online])? 


| | Professional development offerings N = 27, Mean = 3.41, Range: 0-15 


PL12. Please indicate the degree to which you agree or disagree with the following statements. 


LD | Disagree Agree Sine N/A 
disagree agree 
| was able to reach my LDC coach if | had any 2 0 4 24 0 
questions about LDC. (N = 30) (6.7%) 0% (13.3%) | (80.0%) 0% 
(0%) (0%) 
en ie : : eee . 
(N = 30) (0%) (6.7%) | (43.3%) | (36.7%) | (13.3%) 
ie ae ieatmcane | oo 2 7 | 10 : 
school. (N = 30) (0%) (6.7%) (56.7%) (33.3%) (3.3%) 
LDC coaches were able to connect me with 0 il 8 18 3 
additional resources when needed. (N = 30) (0%) (3.3%) (26.7%) | (60.0%) | (10.0%) 
It was challenging to coordinate with our LDC 
. : 5 14 5 4 2 
coach on how to structure Professional Learning 16.7% AG.7% 16.7% 13.3% 6.7% 
Community time. (N = 30) (16.7%) (46.7%) | (16.7%) | (13.3%) (6.7%) 
When | reached out to our LDC coach, he or she 1 0 z 22 0 
responded quickly. (N = 30) (3.3%) (0%) (23.3%) | (73.3%) (0%) 
1 0 6 23 0 
Our LDC coach was easy to work with. (N = 30 
Y APE | fe coy) (0%) | (20.0%) | (76.7%) | (0%) 
Our LDC coach was knowledgeable and 0 0 7 23 0 
provided high quality guidance. (N = 30) (0%) (0%) (23.3%) | (76.7%) (0%) 
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Module Creation 


PL13. During the current school year (2016-17), how many LDC modules did your PLC 
individually or collaboratively adapt from existing modules (e.g., modules created in 
a prior year and/or modules from the LDC Library in CoreTools)? 


| | Adapted modules N =27, Mean = 1.19, Range: 0-4 


PL14. During the current school year (2016-17), how many LDC modules did your PLC create 
(either individually or in a group)? Only include modules built from scratch, not 
those adapted from existing modules in the LDC library. 


| | New modules N = 29, Mean = 2.66, Range: 0-9 


PL15. How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 


(N = 30) 
| | Modules were created by individual teachers. 18 liaisons (60.0%) 
Modules were created by teams of two or more teachers. 13 liaisons (56.7%) 
Modules were created by the PLC as a whole. 1 liaisons (3.3%) 
Other (please specify) 1 liaisons (3.3%) 
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PL16. Please indicate to what extent you were able to do each of the following when 
creating LDC modules and/or mini-tasks. 


Toa 
iota A little bit moderate To etet 
all extent 
extent 
Select a set of focus standards for a writing 0 (0%) 0 (0%) 7 (24.1%) 22 (75.9%) 
‘0 ‘0 . 0 . 0 


assignment (N = 29) 


Create a standards-driven writing assignment 


fackineseh 0(0%) | 1(3.4%) | 6 (20.7%) | 22 (75.9%) 


Select high quality, complex texts and other 
materials that allowed students to engage in 0 (0%) 4 (13.8%) | 10 (34.5%) | 15 (51.7%) 
deeper learning (N = 29) 


Create a writing assignment that provided 
multiple opportunities for students to engage 0 (0%) 3 (10.3%) | 10(34.5%) | 16 (55.2%) 
with the material (N = 29) 


Identify the skills students need to develop in 
order to complete a writing assignment 0 (0%) 0 (0%) 14 (48.3%) | 15 (51.7%) 
(N = 29) 


Create daily lessons to teach the skills a student 


0, fo} Lo} fo} 
needs to complete a writing assignment (N = 29) more) 7 (24.1%) | 10 (34.5%) | 12 (41.4%) 


Plan for a variety of methods to assess student 


fo} fo} fo} fo} 
progress (e.g. mini-task scoring guides) (N = 29) 1 (3.4%) | 7 (24.1%) | 12 (41.4%) 9 (31.0%) 


Make connections to previous or future learning 
that make a writing assignment relevant for 0 (0%) 4 (13.8%) | 14 (48.3%) | 11 (37.9%) 
students (N = 29) 
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School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


PL17. What proportion of PLC meetings focused on LDC did your school administrator attend? 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 
About one half of LDC PLCs 


About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 


(N = 27) 


13 liaisons (48.1%) 
4 liaisons (14.8%) 
5 liaisons (18.5%) 
2 liaisons (7.4%) 

3 liaisons (11.1%) 


PL18. Please indicate the degree to which you agree or disagree with the following statements. 


My school administrator... Strenely Disagree Agree shengly 
disagree agree 
had a firm understanding of LDC. (N = 27) 2 (7.4%) 4 (14.8%) 18 (66.7%) | 3 (11.1%) 
protected common planning time for our LDC ‘i ‘ és 4 
Professional Learning Community to meet. (N = 27) 0 (0%) 3 (11.1%) 16 (59.3%) 8 (29.6%) 
encouraged teachers to participate in LDC. (N = 27) 0 (0%) 2 (7.4%) 16 (59.3%) | 9 (33.3%) 
expressed concerns that implementing LDC was 
taking time away from other instructional priorities. | 4 (14.8%) | 12 (44.4%) | 10 (37.0%) 1 (3.7%) 
(N = 27) 
communicated how using LDC’s tools supported F és js ‘i 
specific school initiatives and/or goals. (N = 27) 1 (3.7%) 3 (11.1%) 17 (63.0%) 6 (22.2%) 
provided me with feedback about my LDC planning ‘ é ‘ ‘ 
and/or instruction, (N=27) 2 (7.4%) 8 (29.6%) 17 (63.0%) 0 (0%) 
made formative assessment a priority at my school. 0 (0%) 1 (3.7%) 21 (77.8%) 5 (18.5%) 


(N = 27) 


104 


Project Liaison Leadership Role 


PL19. Please indicate the degree to which you agree or disagree with the following statements 
about your role in leading your school’s LDC implementation. 


Strongly 


; Strongly 

disagree Disagree Agree agree 
| met regularly with my school administrator to make . : e 4 
planning decisions around LDC. (N = 27) 1 (3.7%) 8 (29.6%) 15 (55.6%) 2 (al) 
| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 0 (0%) 6 (22.2%) 16 (59.3%) | 5 (18.5%) 
(N = 27) 
| was involved in discussions about how to expand LDC . 6 ii - 
implementation at my school in future years. (N = 27) Oe 6 (22.2%) 17 (63.0%) 4 (14.8%) 
My role as a LDC project liaison allowed me to 
effectively advocate for additional resources on my 1 (3.7%) | 10(37.0%) | 14(51.9%) 2 (7.4%) 
campus. (N = 27) 
| was involved in setting instructional goals for the LDC . Z ‘ 4 
workat myschool. (N'=27) 0 (0%) 6 (22.2%) 17 (63.0%) 4 (14.8%) 
| met regularly with my LDC coach to manage the LDC " 6 6 é 
workplan, IN=27) 0 (0%) 2 (7.4%) 15 (55.6%) | 10 (37.0%) 
| feel that my position as an LDC project liaison allowed 
me to build my capacity as an instructional leader 0 (0%) 4 (14.8%) 13 (48.1%) | 10 (37.0%) 


among my colleagues. (N = 27) 
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Alignment 


PL20. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly Bieseree Agree Strongly 

disagree agree 
carnal coed Lc mtemertatentoou | ovo | sanrm | rsissam | ream 
ic aed teachers createwiingasrewmentto | gry) | (69x) | 19465.5%) | 8 27.6% 
neh Re other initiatives taking place in 0 (0%) 4 (13.8%) 18 (62.1%) 7 (24.1%) 
College and career Ready Standarde, (N= 29) | 01%) | 4 (23.8%) | 18(62.1%) | 7 (24.1% 
The im spentimpementng Oe teed wth | 3 a9.3y | sac4ax) | 724.1%) | 5 072% 
oc haped pepaestientenmy shoots Ta esa | sca7zx) | 17 686%) | 6(207% 
toes cic or tare ffnugon oc scrus | ovom) | sz7ex) | r4ueax) | 724% 
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Scale-up and Sustainability 


PL21. Please indicate the degree to which you agree or disagree with the following statements. 


Stronely Disagree Agree Stance ly 
disagree agree 


| expect that most teachers participating in 
LDC this year will continue to do so next year. 2 (6.9%) 3 (10.3%) 20 (69.0%) 4 (13.8%) 
(N = 29) 


Teachers at my school who were not part of 
the LDC PLC meetings used the LDC planning 8 (27.6%) 15 (51.7%) 5 (17.2%) 1 (3.4%) 
process and/or LDC CoreTools. (N = 29) 


As a result of LDC, new collaborations across 
grades and/or subjects were created or are 3 (10.3%) 15 (51.7%) 9 (31.0%) 2 (6.9%) 
being launched at my school. (N = 29) 


Teachers and administrators at my school are 
committed to sustaining the LDC initiative. 2 (6.9%) 5 (17.2%) 16 (55.2%) 6 (20.7%) 
(N = 29) 


| expect our LDC PLC to increase in size next 


sear Naas) 1 (3.4%) | 10(34.5%) | 12 (41.4%) | 6 (20.7%) 


District Support 


PL22. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly sioseice Fores Strongly Don’t 
disagree agree know 
District leaders supported the implementation 2 2 5 6 14 
of LDC. (N = 29) (6.9%) (6.9%) (17.2%) | (20.7%) (48.3%) 
District leaders had a firm understanding 1 2 2 7 17 
of LDC. (N = 29) (3.4%) (6.9%) (6.9%) (24.1%) (58.6%) 
District leaders are interested in spreading the 2 2 2 6 17 
use of LDC to additional schools. (N = 29) (6.9%) (6.9%) (6.9%) | (20.7%) (58.6%) 
District professional development efforts were 2 1 5 4 17 
aligned with the LDC initiative. (N = 29) (6.9%) (3.4%) | (17.2%) | (13.8%) (58.6%) 
District leaders visited my school to discuss the 2 3 5 6 13 
implementation of LDC. (N = 29) (6.9%) (10.3%) | (17.2%) | (20.7%) (44.8%) 
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Ke) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 


O Zoom meetings, written feedback on teacher work in LDC CoreTools, Teaching Channel 
discussions, emails, etc. 


e /n-person coaching 


O Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


PL23. What supports did you find the most useful and why? 


PL24. What supports were not helpful and why? 


PL25. In what ways could LDC implementation be improved in your school in the future? 
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Appendix D: 
Administrator Survey and Responses 


2016-2017 LDC School Administrator Survey: NYCDOE 


i) LDC Participation 


A1. What is your role at the school? 


(N = 32) 
Principal 18 admins (56.3%) 
Assistant Principal 13 admins (40.6%) 
|__| Other (please specify) 1 admins (3.1%) 


Professional Learning Community 


A2. What proportion of LDC Professional Learning Community (PLC) meetings did you attend 
during the current school year? 


(N = 32) 
Less than one quarter of LDC PLCs 16 admins (43.8%) 
About one quarter of LDC PLCs 6 admins (18.8%) 
About one half of LDC PLCs 4 admins (12.5%) 
About three quarters of LDC PLCs 4 admins (12.5%) 
More than three quarters of LDC PLCs 4 admins (12.5%) 
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LDC Training and Support 


A3. How many in-person and/or online LDC professional development offerings for school 
administrators and project liaisons did you attend during the current school year (e.g., 


Launch Day [in person], February Administrator Meeting [in person], Bi-Monthly 
Community Meetings [online])? 


| | LDC Professional development offerings N = 30, Mean = 2.67, Range: 0-10 


A4. Please indicate the degree to which you agree or disagree with the following statements. 


Stoney Disagree Agree SHOpely N/A 
disagree agree 
| was able to reach LDC staff when | had P é F ‘ 
questions about WDC; (N-=30) 0 (0%) 0 (0%) 9 (30.0%) 19 (63.3%) 2 (6.7%) 
My school has adequate technology to " - ‘ P 
access LDC online resources. (N = 30) 0 (0%) 1 (3.3%) 15 (43.3%) | 16 (53.3%) 0 (0%) 
LDC offered sufficient professional 
development opportunities for LDC 0 (0%) 1 (3.3%) | 20 (66.7%) 9 (30.0%) 0 (0%) 
project liaisons. (N = 30) 
LDC offered sufficient professional 
development opportunities for school 0 (0%) 2 (6.7%) | 19 (63.3%) 8 (26.7%) 1 (3.3%) 
administrators. (N = 30) 
LDC staff members were able to connect 
me with additional resources when 0 (0%) 1 (3.3%) | 16(53.3%) | 10 (33.3%) | 3 (10.0%) 


needed. (N = 30) 
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Classroom Observation 


A5. How many times during the current school year did you observe teachers implementing 
an LDC module? 


(N = 29) 
0 times ib Skip to A7 1 admins (3.4%) 
1-2 times 10 admins (34.5%) 
3-5 times 13 admins (44.8%) 
| | 6or more times 5 admins (17.2%) 


A6. On average, how effective were LDC modules in developing students’ literacy skills? 


(N = 28) 
Not effective 0 admins (0%) 
| ] A little effective 2 admins (7.1%) 
Moderately effective 20 admins (71.4%) 
[ Very effective 6 admins (21.4%) 
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Impact on Teacher Practice 


A7. Based on your oversight of the LDC program, please indicate on average how much the 


teaching practice of LDC PLC members improved in each of the following areas: 


Not at all A little Moderately | A great deal 
Selecting a set of focus standards for a writing P is ‘ F 
assignment (N = 28) 0 (0%) 0 (0%) 18 (64.3%) 10 (35.7%) 
Creating standards-driven writing tasks of sufficient 
quality and scope to allow students to think deeply . o . " 
about disciplinary content and practice literacy skills 0 (0%) 2 (7.1%) 14 (50.0%) 12 (42.9%) 
(N = 28) 
Identifying the skills students need to develop in 7 . é e 
order to complete a writing assignment (N = 28) 0 (0%) 0 (0%) 19 (67.9%) 9 (32.1%) 
Creating daily lessons to teach each skill a student é e 6 z 
needs to complete a writing assignment (N = 28) 1 (3.6%) 6 (21.4%) 13 (46.4%) 8 (28.6%) 
Assessing students’ progress as they work toward . ‘ is _ 
completing a writing assignment (N = 28) 0 (0%) 5 (17.9%) 15 (53.6%) 8 (28.6%) 
Tracking and analyzing evidence about student 2 (7.1%) 7 (25.0%) 14 (50.0%) 5 (17.9%) 
progress in a systematic way (N = 28) 
Identifying patterns of student understandings or 0 (0%) 10 (35.7%) 9 (32.1%) 9 (32.1%) 
misconceptions (N = 28) 
Using evidence of student performance of standards P é ; : 
to shape future instructional decisions (N = 28) 0 (0%) 8 (28.6%) 12 (42.9%) 8 (28.6%) 
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Impact on Student Learning 


A8. Please indicate the degree to which you agree or disagree with the statements below: 


Strongly Disses Reee Strongly 
disagree agree 
‘ie ae students develop reading skills. 0 (0%) 1 (3.6%) 22 (78.6%) 5 (17.9%) 
LDC ffective in i i tudents’ tent 
cee ia ai 0 (0%) 2 (7.1%) 19 (67.9%) 7 (25.0%) 
LDC modules effectively supported students in 5 é 5 
completing writing assignments. (N = 28) 0 (0%) 1 (3.6%) 19 (67.9%) 8 (28.67%) 
LDC was effective in improving the quality of 
students’ writing. (N = 28) 0 (0%) 2 (7.1%) 18 (64.3%) 8 (28.6%) 
LDC supported students’ development of skills 0 (0%) 1 (3.6%) 21 (75.0%) 6 (21.4%) 


needed for college and career readiness. (N = 28) 


LDC increased students’ capacity to analyze and 
understand the components of a writing 0 (0%) 2 (7.1%) 19 (67.9%) 7 (25.0%) 
assignment task. (N = 28) 


LDC helped students develop speaking and 


iceching celle: (aes 2 (7.1%) 2 (7.1%) | 17(60.7%) | 7 (25.0%) 


Overall, LDC helped improve students’ literacy 


cach cece ue 0 (0%) 2 (7.1%) | 18 (64.3%) | 8 (28.6%) 
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¥,) Administrator Leadership Role 


A9. Please indicate the degree to which you agree or disagree with the following statements 
about your role in leading LDC implementation in your school: 


Strongly Disagree Agree StrOnely 
disagree agree 


| was able to shape LDC implementation at my school. 


(N = 28) 0 (0%) 3 (10.7%) | 20(71.4%) | 5 (17.9%) 


| met regularly with the LDC project liaison in my 
school to stay abreast of implementation progress. 0 (0%) 2 (7.7%) | 20(76.9%) | 4(15.4%) 
(N = 26) 


| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 0 (0%) 5 (17.9%) | 19 (67.9%) | 4(14.3%) 
(N = 28) 


| led discussions about how to expand my school’s LDC 


10) 9 io) fe} 
implementation in future years. (N = 28) 1 (3.6%) 7 (25.0%) | 17 (60.7%) | 3(10.7%) 


| made changes to school schedules to accommodate 


fo} 0, fo} 0, 
LDC professional learning time. (N = 27) 0 (0%) 1 (3.7%) 19 (70.4%) | 7 (25.9%) 
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Alignment 


A10. Please indicate the degree to which you agree or disagree with the following 


statements: 

Strenely Disagree Agree Strongly 

disagree agree 
oe eageratontoowr | oom | 186% | 191679% | 8 as6m 
crepes eres ene neasignmentto | oom | rem | 191643 | 924m 
mnie ae other initiatives taking place in 0 (0%) 2 (7.1%) 17 (60.7%) | 9 (32.1%) 
te mas tates forimotemertng my stes | gon) | 340.7%) | 29,679%) | 6 (21.4% 
other important initiates at my school (N= 28) | 4(143%) | 12(42.9%) | 10(35.7%) | 27.196 
oohienunns ees in my school for current 0 (0%) 1 (3.6%) 21 (75.0%) | 6 (21.4%) 
twas dif for teachers fofonon Loctacass | g 14.3) | 10135.) | 13 464%) | 1 (3.6% 
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Scale-up and Sustainability 


A11. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly Seeqiee Reree Strongly Don’t 
disagree agree know 
| expect that most teachers participating 
in LDC this year will continue to do so 1 (3.6%) 0 (0%) 17 (60.7%) | 8(28.6%) | 2 (7.1%) 
next year. (N = 28) 
Teachers at my school who were not 
part of the LDC PLC meetings used the é 4 ‘ 7 4 
bbe plantains pracesé and/or lc 5 (17.9%) | 10 (35.7%) 8 (28.6%) 3 (10.7%) 2 (7.1%) 
CoreTools. (N = 28) 
As a result of LDC, new collaborations 
across grades and/or subjects were ‘ 4 - 4 
createdoware beine laiiched at my 1 (3.6%) 3 (10.7%) 17 (60.7%) | 5 (17.9%) 2 (7.1%) 
school. (N = 28) 
Teachers and administrators at my 
school are committed to sustaining the 1 (3.6%) 0 (0%) 17 (60.7%) | 9 (32.1%) | 1 (3.6%) 
LDC initiative. (N = 28) 
| expect our LDC PLC to increase in size é ‘ ‘ és " 
next vear ANSoe 2 (7.1%) 3 (10.7%) 13 (46.4%) | 8 (28.6%) 2 (7.1%) 
District Support 
A12. Please indicate the degree to which you agree or disagree with the following 
statements: 
Shonely Disagree Agree Strongly Don’t know 
disagree agree 
District leaders supported the ‘ . 7 5 4 
implementation of LDC. (N= 28) 2 (7.1%) 1 (3.6%) 13 (46.4%) | 3(10.7%) | 9 (32.1%) 
District leaders had a firm understanding - js . ‘ ; 
of LDC. (N = 28) 2 (7.1%) | 4(14.3%) | 9 (32.1%) | 3 (10.7%) | 10 (35.7%) 
District leaders are interested in spreading ‘ “i 5 ‘ j 
the use of LDC to additional schools. (N = 28) 1 (3.6%) 1 (3.6%) 1D (35.7%) | 310.7%) | 1346.47) 
District professional development efforts " ; és " _ 
were aligned with the LDC initiative. (N =28) 2 (7.1%) | 5 (17.9%) | 8 (28.6%) | 3 (10.7%) | 10 (35.7%) 
District leaders visited my school to discuss js ‘i . ‘ ‘ 
the implementation of LDC. (N = 28) 3 (10.7%) | 6 (21.4%) | 8 (28.6%) | 3(10.7%) | 8 (28.6%) 
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1 1 Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 


O Zoom meetings, written feedback on teacher work in LDC CoreTools, Teaching Channel 
discussions, emails, etc. 


e /n-person coaching 


O Summer training, in-person support visits from LDC and District Lead, in-person professional 
development opportunities, etc. 


A13. What supports did you find the most useful and why? 


A14. What supports were not helpful and why? 


A15. In what ways could LDC implementation be improved in your school in the future? 
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Appendix E: 
Outcome Analysis Methodology 


Analysis Model Specification 


For our outcome analyses, we used a threshold of p < .05 to determine whether there was 
a Statistically significant impact of LDC on ELA achievement. In addition to the LDC treatment 
indicator, a teacher effect for years of experience was included, as well as an aggregate 
indicator measuring the mean prior performance of each student’s classroom peers. The fixed 
effects also included student characteristics to identify the matched comparison sample of 
students, such as baseline achievement, socioeconomic status, demographics, language 
proficiency, grade level, and participation in special education. 


The three-level MMMC model was used to estimate the impacts of the LDC intervention 
on student learning. This same analytic model will be used to estimate impacts in future years. 
The general specification for the middle school level MMMC model is shown in the following 
equation using similar notation proposed by Browne et al. (2001, equation 6) and applied in 
Tranmer, Steel, and Browne (2014, equation 3). 


aN al (3) (2) 
Vi = XiB + Uschoon(i >: wijuy +e 
j€Teacher(i) 


i = 1,..,n Teacher(i) ¢ (1,...,J) 


(3) 2 (2) 2 
Uschool(i) ~ N(0, oxy), uj ~ N(0, ony), e, ~ N(0,2/) 


In this model y; is the student achievement score response, Xiis a vector of the fixed 
covariates and f is the vector of the corresponding fixed effects. School(i) is the school which 
student i attends, thus the term ees represents the random effects for that level of 
classification. Within the term > jereacner(i) wij” ; a is the set of j random effects for the 
teachers included in the selected dataset, and w, ; is the weight which sums to 1 for each 
student applied in proportion to the instruction time assigned with each teacher. The following 
presents an example of the full model middle school specification. 


achievement; = By + LDC; * 6, + priorELA; * B2 + priorMath, * B3 + Female; * 6, + LEP; 
* Bs; + SES; * Bg + Black; * B; + Hispanic; * Bg + Asian; * By + OtherEth; 
* Big + SPED; * By, + Grade7; * Biz + Grade8; * Bi3 + Gifted; * By4 
+ priorClassELA; * B15 + TeachExp; * B16 


+ TotalContent; * By7 + Sern w, ur” 


i +e; 


j€Teacher(i) 


(3) (2) 
Uschool(i) ~ N(0, ox), uy ~ N(0, oy), e, ~ N(0,02/) 
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In this model, achievement; is the standardized ELA outcome score for student i; In the 
dosage-dependent model LDC; is the proportion of core class instruction time taken with an 
LDC teacher (ranges 0-1); In the dosage-independent model LDC; is coded as zero for 
comparison students and as 1 for students receiving any level of LDC teacher exposure. 


Female;, LEP;, Black;, Hispanic; , Asian, , OtherEth; SPED,, Gifted;,and SES; are 
student demographic indicators coded 1 if the status is present and 0 if absent; 


Grade7,; and Grade8,; are dummy coded 1 when the grade was attended and otherwise 
as O, allowing Grade 6 students to serve as the reference group. In the models that combine 
elementary and middle school students the additional dummy coded grade variables will be 
included. The elementary school specification would be very similar, with different dummy 
variables for grades; 


priorELA, , and priorMath, are standardized student achievement scores from the prior 
year; 


priorClassELA; is the aggregated mean of the prior ELA scores for all the core class 
peers of student i; 


TeachExp;, is the aggregated mean of the years of teaching experience for those teachers 
which student i was exposed to in her core classes; 


TotalContent; is the aggregated mean weeks of the total ELA, social studies and science 
content enrolled preceding the NYS assessment date which student i was exposed to; 


B, is the impact of LDC, the treatment; 

By2 is the average difference between Cohort 1 and Cohort 2; 
PB & B3 are the effects of the prior score covariates; 

By ... B14 are the effects of the demographic covariates; 


Bis --- P17 are the effects of the aggregated class level covariates 


are a, e; are the error components at the school, teacher, and student level 


respectively assumed to all have mean 0 and variance, OL(2)) ern og respectively. 


Student/Teacher Course Exposure Weighting 


Tables E1 and E2 demonstrate how the process of calculating general MMMC teacher 
weights and LDC treatment weights (for the dosage modeling approach) was conducted 
respectively for elementary and middle school. In elementary school, in the event that a 
student was exposed to more than one teacher, each content area was given equal weight in 
distributing teacher/student exposure. For example, if a student was enrolled for both ELA and 
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social studies/history under one teacher, then that teacher was coded as .67 for having 
contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead each core content 
area exposure contributed to a core content area total sum which formed the basis from which 
the weights were proportioned. Most commonly a student had equivalent days of core 
instruction exposure in each of the three content areas (often 214 days in each content area). 
In that scenario, if a student had exposure to three different teachers, then each teacher would 
contribute one third (.33) of the overall core curriculum exposure and all other teachers in the 
sample would be coded as zero. However, in addition to the typical core science course, extra 
core science courses were also included in the LDC analysis (for example a Grade 8 student 
taking biology), which made it possible then for a student to accumulate more units in science 
than in the other two content areas. The weighting in middle school was always distributed as a 
proportion of the total exposure days in the three content areas. Therefore, if a student 
accumulated 300 science days (across two courses), 200 social studies days, and 200 ELA days, 
the base number of instruction days would be 700 days. If, using that same scenario, the same 
teacher taught both the typical core and biology courses then that teacher would contribute 
three-sevenths (.43) of the overall core curriculum exposure with the social studies and science 
teachers contributing two-sevenths (.285) each, again resulting in the student’s exposure 
adding to a unity (1). 
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Table E1 
Elementary School Student/Teacher Weighting Based on Course Links 


Example student/teacher weighting for use in Example LDC 


Core content MMMC treatment 
area NYSED course name (Weight=Subject Days/Total Days) dosage weight 
ELA Language Arts (Grade Student enrolled 214 days Language Arts Weight=.333 
4,5) (Grade 5) 


with an intervention Teacher: 
Weight=(214/214)*.333=0.333 


Social studies Social Studies (Grade Student enrolled 214 days of core Social Weight=.0 
4,5) Studies (Grade 5) with a non-intervention 
teacher: 


Weight=(214/214) *.333=0.333 


Science Science (Grade 4,5) Student enrolled 214 days Science (Grade 5): Weight=.333 
Weight=(214/214) *.333=0.333 

Total Unity: for every student the student/teacher | Treatment 
weights sum to 1 Weight=.667 


Note. “Days” refers to core content enrolled days preceding the NYS assessment date. 
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Table E2 
Middle School Student/Teacher Weighting Based on Course Mark Links 


Example student/teacher weighting for use in Example LDC 


Core content MMMC treatment 
area NYSED course name (Weight=Subject Days/Total Days) dosage weight 
ELA Language Arts (Grade Student enrolled 214 days Language Arts Weight=.285 
6,7,8) (Grade 8) 


with an intervention Teacher: 
Weight=200/700=0.285 


Social studies Social Studies (Grade Student enrolled 214 days of core Social Weight=.0 
6,7,8) Studies (grade 8) with a non-intervention 
teacher: 


Weight=200/700=0.286 


Science Science (Grade 6,7,8) Student enrolled 200 days Science (grade8) and Weight=.0 
100 days with a non-intervention teacher: 

Also: Weight=374/802=0.430 

Earth Science 

Biology 

Life & Phys Sci— 

Other 
Other ELA, All other Courses Not included in weighting, but used at student Teachers not 
Social studies, level in matching and regression. included in 
and Science analysis 
Total Unity: for every student the student/teacher | Treatment 


weights sum to 1 Weight=.285 


Note. “Days” refers to core content enrolled days preceding the NYS assessment date. 


Calculation of Effect Size 


We calculated student-level effect sizes according to the WWC 3.0 criteria. Specifically, for 
the impact analysis with treatment status as a dichotomous variable, we calculated Hedges’ g, 
the difference in adjusted mean outcomes for the groups divided by the unadjusted pooled 
within-group standard deviation of the outcome measure in the sample, for all outcomes. The 
difference in adjusted mean outcomes is estimated by £, in the models defined above, as 
outlined in the WWC standards handbook for computing effect sizes in multi-level frameworks. 
Specifically, 
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WY 
8 ——— SS 
(n; — 1)s? + (n, - 1)s?2 
nitn, — 2 


where y is £, which is the coefficient from the MMMC for the intervention effect. 


Note that in our analyses the outcome measure is standardized within the analytical 
sample (mean=0, standard deviation=1). As a result, we expected that g would likely be quite 
similar to the B, coefficient from the MMMC Model in the large samples we plan to collect later 
in the study. 


Though it is not standard to use g with a continuous treatment effect, as in the case of 
our dosage-dependent treatment measure, we have defined and matched populations n; and 
n, where treated students (n;) could have any positive treatment value<=1, and control 
students (n,) had a treatment value of zero. We could therefore calculate g in the case of our 
dosage-dependent treatment measure, and again expect that it would not differ substantially 
from the f, coefficient. It is crucial, however, to note that g and 3, in the dosage-dependent 
models reflect the effect size projected for a student who would receive exposure to 
intervention teachers in all of their core classes. Along with this effect we report average 
dosage received by treated students so that the average treatment effect on treated students 
could be calculated. 
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